Aether-7B-5Attn: A 100% Open-Source Sovereign Foundation Model — and a Controlled Experiment in Heterogeneous Attention

VIDRAFT_LAB

SeaWolf-AI

Follow

FINAL-Bench

Not just the weights. The data recipe, the training code, the full logs, the intermediate checkpoints, and the complete architecture source — everything needed to rebuild a foundation model from scratch, released under Apache-2.0. Built from scratch by a single Korean AI startup, VIDRAFT.

Most "open" language models are open in name only: you get a bag of weights and a permissive license, but the data that made them, the code that trained them, and the decisions along the way stay locked away. You can run such a model. You cannot audit it, reproduce it, or rebuild it — and therefore you cannot truly own it.

Aether is built on the opposite premise. It follows the "fully open" standard that Allen Institute's OLMo established — weights plus data recipe plus training code plus logs plus checkpoints — and pairs it with a genuine architectural research question: does it help to give different transformer layers different attention mechanisms, and can we measure that cleanly? This article walks through both the engineering and the science.

1. At a glance

Aether-7B-5Attn is a decoder-only, 6.59B-parameter Mixture-of-Experts language model (~2.98B active per token) whose 49 layers place five distinct attention mechanisms on a 7×7 Latin square. It was trained from scratch on 144.2B tokens using 16× NVIDIA B200 GPUs, and released fully open under Apache-2.0.

2. The thesis: sovereignty is reproducibility, not download access

"Sovereign AI" is often reduced to "a model in our language, on our servers." But downloading someone else's weights is not sovereignty — it is tenancy. A model whose training data is undisclosed is a black box you can neither verify for contamination and bias, nor patch at the source, nor regrow when it drifts.

Real sovereignty is the capacity to rebuild. It requires the recipe, not just the dish. Aether is engineered so that the entire pipeline — the exact public data sources and mixing weights, the tokenizer, the FSDP launcher and training loop, every hyperparameter, and the step-by-step logs — is in your hands. Run the recipe and the training binaries reproduce byte-for-byte.

This is deliberately the OLMo-tier bar. Our claim is not that this openness is unique — OLMo (USA) and Apertus (Switzerland) meet it too. Our claim is narrower and verifiable: among the world's fully-open sovereign foundation models, Aether is the only one built by a single AI startup rather than a national lab, a government consortium, or a university coalition — and it is the first Korean model to open its training data and training code.

3. Architecture I — why should every layer run the same attention?

Since GPT, nearly every transformer has used the same attention operator in every layer. A 49-layer model runs the identical operation 49 times. Yet layers demonstrably do different work: early layers resolve local morphology and grammar, middle layers assemble syntactic structure, late layers reason across the whole document. If the job differs by depth, why must the operator be identical at every depth?

This is a question about an unexamined convention, not about squeezing out a benchmark point. "Same attention everywhere" was never a proven optimum — it was simply never seriously challenged. Aether relaxes the assumption and treats attention type as a per-layer architectural variable.

The motivation is that each attention mechanism carries a different inductive bias and a different cost, and none strictly dominates:

If no single mechanism wins everywhere, then choosing one and repeating it 49 times also repeats its weakness 49 times. Heterogeneity is a way to spend each layer's compute on the inductive bias that layer can best use.

4. Architecture II — seven labels, five mechanisms

Aether defines seven attention labels, each occupying exactly 7 of the 49 layers (7 × 7 = 49):

We are precise about this on purpose: although there are seven labels, they are realized by five distinct computational mechanisms in this release (compress currently collapses onto the full path and linear onto a gated SDPA). Hence the honest name — 5Attn, not 7Attn. The attention operators themselves are variants of published ideas (differential attention, native sparse attention, sliding-window attention); our contribution is not a new attention operator — it is the controlled placement.

5. Architecture III — the Latin square as a control, not decoration

"Mixing should help" is an easy intuition and a treacherous one. Mix carelessly and the result becomes impossible to attribute: if full happens to cluster in the late layers, the model did well "because its late layers are full," not "because it is heterogeneous." The confound of depth swallows the signal.

So Aether's 49 layers are not shuffled — they form a 7×7 Latin square. Indexing each layer by (row = layer // 7, col = layer % 7), its attention type is type = (row + col) mod 7. Because 7 is prime, this cyclic construction is a proper Latin square with a hard guarantee: every attention type appears exactly once in every row and exactly once in every column — i.e., exactly once at each depth band and each phase position. No type can concentrate at any depth.

That guarantee is the whole point. The Latin square converts a vague design choice into a controlled experiment: it isolates the effect of heterogeneous placement from the effect of which layer got which operator, so ablations against a homogeneous baseline are attributable rather than anecdotal. To our knowledge, no released model arranges heterogeneous attention on a Latin square — and even if that is someday shown false, the design still stands on its own experimental merits.

6. Architecture IV — the Mixture-of-Experts

Each layer's feed-forward block is a fine-grained MoE: 25 experts with top-7 routing plus 1 always-on shared expert, trained with an auxiliary router load-balancing loss to keep expert utilization even. The shared expert captures common computation while the routed experts specialize; only ~2.98B of the 6.59B parameters activate per token, giving the capacity of a larger model at the FLOPs of a smaller one. This fine-grained-plus-shared design is also, candidly, part of why efficiency is hard (see §9).

7. Training — 144.2B tokens, transparently

Aether was pretrained from scratch on 144.2B tokens over 162,000 steps, on 16× NVIDIA B200 in a 2-node FSDP configuration, across roughly 46 days. The released base is the annealed 162k checkpoint; the pre-anneal 110k / 115k / 162k checkpoints are published separately as a dataset so that training-dynamics research — loss-landscape evolution, capability emergence, checkpoint interpolation — is possible on a fully-open model.

The data mixture is math-forward by design, to build a reasoning substrate before breadth:

Every source is a public repository, and the mixing weights, tokenizer, and EOS handling are published with the recipe — the property that makes the run byte-reproducible.

8. The instruct model — chosen by held-out accuracy, not by loss

Aether-7B-5Attn-it is the SFT variant for multiple-choice / benchmark-style answering, and it demonstrates a methodological point we consider important: for small models, training loss is the wrong checkpoint selector. A higher learning rate reliably drives loss down while driving real capability down with it — loss rewards format-fitting, not competence.

So we swept three learning rates (2e-6 / 6e-6 / 2e-5) and selected on held-out benchmark accuracy, on sets disjoint from the SFT data (MMLU-auxiliary):

• 6e-6 (selected): K-AI 4-benchmark average 34.9% — best and balanced.
• vs. base before SFT: 26.7% → 34.9% (+8.2pp), same held-out set, same harness.
• 2e-5: lowest training loss, but one subject collapsed — the textbook signature of format degradation.

9. Honest limitations

Openness includes disclosing the rough edges:

• No fast KV cache in this build. The nsa branch addresses its key/value cache with custom per-branch indices that the stock DynamicCache does not provide, so cached autoregressive decoding is not wired up in the released code. Generation therefore runs without a KV cache (the live demo streams tokens so you can watch progress). Fast serving is a separate engineering track.
• Low MFU is expected here, not a bug. Fine-grained MoE under naive FSDP is a known low-utilization regime; our numbers sit in that documented band. We report it openly rather than hiding it.
• Scale. 6.59B is small. Aether is not positioned as a leaderboard champion — it is a reproducible reference implementation of a sovereign foundation model and a clean testbed for the attention-placement question.
• batch_size = 1 only. Some attention branches (the NSA family) ignore padding masks, so batching padded sequences can silently corrupt results.

10. Where Aether sits among sovereign fully-open models

An honest roll-call of the world's fully-open (data + code + logs) sovereign foundation models:

• 🇯🇵 LLM-jp (NII) — the first of this class, 2023-10
• 🇫🇮 Viking (Silo AI)
• 🇨🇭 Apertus (EPFL + ETH + CSCS)
• 🇺🇸 OLMo 3 (Allen Institute, non-profit)
• 🇩🇪 Soofi (Germany)
• 🇰🇷 Aether (VIDRAFT)

The two differentiators we can actually verify — and only these two:

1. Aether is the only one of the six built by a single AI startup. The others are national labs, government consortia, or university coalitions. This is a statement about capital efficiency and access: building — not merely fine-tuning — a fully-open foundation model is no longer the exclusive domain of national-scale institutions.
2. Aether carries the most attention types (five), placed in a Latin-square heterogeneous layout. Everyone else ships a single, homogeneous attention.

We do not claim data or license openness as a solo advantage — it is on par with OLMo and Apertus. And because Japan shipped first, we claim only first Korean, never world-first. Restraint in claims is, we think, part of doing open science properly.

11. Why it matters

Aether is a small model that makes a large point in three ways.

• For sovereignty: it is an existence proof that a mid-size organization can hold the entire means of production of a foundation model — recipe, code, logs, checkpoints — and therefore rebuild and evolve it independently. That capability, not a downloaded checkpoint, is what "sovereign AI" should mean.
• For science: the Latin square turns an intuition ("mix the attentions") into a measurable, attributable research instrument, and the released checkpoints and logs let anyone study the training dynamics of a heterogeneous-attention MoE end to end.
• For the ecosystem: every artifact is Apache-2.0 and reproducible, so this is not a demo to admire but a starting point to fork — a foundation others can stand on to build their own.

We will not claim Aether is a special model. What we will claim is this: a single startup built a foundation model from scratch, placed five attention mechanisms on a Latin square as a controlled experiment, and released every last piece of it, openly. If this repository lets you start your own model, it has done its job.

Links

• 🏢 Organization: VIDRAFT / FINAL-Bench
• 🧬 Collection: Aether Foundation Model
• 📦 5 Attention Base model: Aether-7B-5Attn
• 🎯 5 Attention Instruction-tuned: Aether-7B-5Attn-it
• 🧩 5 Attention Intermediate checkpoints: Aether-7B-5Attn-checkpoints
• 🚀 5 Attention Live demo (Space): Aether-Sovereign-AI
• 📦 7 Attention Base model: Aether-7B-7Attn
• 📦 11 Attention Base model: Aether-6B-11Attn
• 🖼️ Figures: hero · Latin-square attention layout · six-country comparison

Built by VIDRAFT (주식회사 비드래프트) · License: Apache-2.0