Barbados — a looped MoE research prototype

This is a research prototype. It is not a useful general-purpose assistant. Its saved generations are repetitive, off task and sometimes incorrect. Useful instruction following, coding, broad reasoning gains and acceleration are unestablished. This release documents a completed training trajectory and its measured cache–quality tradeoff.

The latest findings brief collects the completed gate-policy interaction, matched continuation control and gates-only qualification results.

Barbados is a 112,237,056-parameter causal language model with a backbone trained from random initialization, three recurrent passes, MELT-style KV sharing and small visible-response fine-tuning. This independent research has no affiliation with or endorsement by Mistral AI. Barbados is named after the Barbados threadsnake.

Architecture

Component	Implemented setting
Physical stack	Six blocks, hidden width 384
Recurrence	Three complete passes; 18 block visits with shared attention/expert weights
Attention	Grouped-query attention: six query heads, three KV heads, head dimension 64; RoPE
Experts per block	24 routed experts and two always-active shared experts; intermediate width 384
Expert selection	Four routed experts plus both shared experts per token; separate learned routers per loop
Tokenizer	Reused SmolLM tokenizer, vocabulary 49,152, EOS ID 0
Context	At most 1,023 tokens, including formatted prompt and continuation
Parameter count	110,465,280 backbone parameters plus 1,771,776 latent-memory gate parameters

Sparse expert selection does not skip recurrent passes: all three loops execute. The geometry and native recipe derive from the public LOOM implementation at commit a2ee5a7, with reduced size, loop count, global batch and training exposure. This is a budget derivative, not a published LOOM/Ouro run reproduction. Looping, sparse experts and MELT sharing are prior methods; no invention is claimed here.

Each physical attention layer has one latent-memory gate shared across its three visits: six gates stored in 18 learned tensors. For normalized attention input u, loop one initializes m = u. Later loops compute z = sigmoid(Wu*u + Wm*m_previous + b) and m = z*m_previous + (1-z)*u. Queries use u; keys and values use m. Gate initialization is Wu = Wm = 0, b = -4.

Fixed two-unit CPU diagnostics measured FP32 mean retention of 1.02–1.66% FULL, 1.90–4.22% SHARED, and 1.94–4.49% after response SFT, versus 1.80% at initialization. FULL pre-activation means shifted below −4 in all 12 layer/loop cells; SHARED and the child shifted above −4 in ten cells and below in two, with wider distributions. Both mean and spread matter.

A completed policy-by-gate 2×2 on the pre-SFT SHARED@32 parent and 32,065 pilot targets finds a conditional benefit under SHARED and a cost under FULL. Resetting the gates raises SHARED CPU FP32 CE 3.724062 → 3.729816, but lowers FULL CE 3.708169 → 3.695786. Learned gates help SHARED in chunks 3–20 and hurt FULL in every chunk; the policy interaction is +0.018137 nat/target, with exact chunk-zero controls. This supports sharing-dependent compensation on this coadapted backbone. It does not reproduce training with fixed gates or identify a unique adaptation mechanism. The final SFT child's gate contribution has not been ablated. See the new control report and chunk figure and earlier gate statistics.

The SHARED policy retains a complete final-loop KV prefix and the latest 50 positions for each earlier loop, separately at each physical layer. An earlier-loop query at position i uses final-loop KV for positions j < max(0, i-50) and its own loop's KV for more recent positions. All selected positions enter one causal softmax. Chunk size and window are both 50. Chunks commit after all loops finish; old final-loop states therefore come from completed chunks. Sharing operates across loops at the same layer.

Training keeps full gradients through all loops and selected earlier chunks; history is not detached. Residual carry and latent slots reset between chunks; explicit KV preserves token history. Cache savings do not imply equivalent autograd-memory savings.

Training lineage

Stage	Retained training	Initialization and objective
Native backbone	7,217 updates; 920,770,653 next-token targets	Random weights, seed 0; final-loop causal CE
MELT-style adaptation	32 updates; 4,087,238 replay targets per arm	Identical gated native states for FULL and SHARED; teacher-free final-loop CE
Released SHARED child	Three epochs; 21 updates; 147,597 supervised response-target exposures	Saved SHARED parent; fresh FP32 Adam; response-only CE

No pretrained foundation weights or teacher were used for the backbone. Scratch pretraining uses one prepared pass over FineWeb-Edu documents from two shards of revision 87f09149ef4734204d70ed1d046ddc9ca3f2b8f9, component-based train/development separation and approximate deduplication. It uses the author LeCun initializer, microbatch 16, accumulation eight, global batch 128, AdamW (0.9, 0.95), peak LR 4e-4, weight decay 0.1, clip norm 1 and a 10% warmup / 80% stable / 10% decay schedule to 10% of peak. Native capacity factor is 1.5. Adaptation and causal evaluation disable capacity dropping to avoid future-token competition for expert slots. The released source's configured auxiliary z-loss remains disconnected from the active objective.

The MELT comparison replays the same FineWeb-Edu prefix for both arms. FULL retains distinct complete histories for all loops; SHARED uses the policy above. Both use active gates, full gradients and the same 32-update budget, with zero skipped updates. This is a reduced, teacher-free CE adaptation of MELT, not its complete teacher-supervision objective, sequence length or training budget. BF16 parameter views remain active through forward and backward, with FP32 masters, gradients, optimizer states and router projection arithmetic.

Small Fable visible-response SFT

The final stage uses public, uploader-attributed responses from TheFusionCube/Fable-5-CoT-Traces, revision 3624e3a8aab36a89dc3b89aebef0e034a64ea04e. The generating model's identity is not independently authenticated; these are visible answers, not authenticated hidden/private chain-of-thought. Selection excludes decoys, ambiguous metadata, truncation, unsuitable tasks and overlength examples. Answers remain verbatim, with unverified correctness. The duplicate armand0e chat conversion adds no data.

Split	Unique examples	Task families	Input tokens per epoch	Supervised targets per epoch
Train	68	19	54,651	49,199
Validation	5	3	4,041	3,663
Test	12	3	9,369	8,455

Question families, linked sessions and normalized duplicate payloads are grouped before splitting. Prompt and response are separately tokenized, concatenated, and followed by EOS. Only response tokens and EOS receive CE labels; prompt-state gradients remain. The exact wrapper is:

### Context
{prompt}

### Response

Three epochs reuse the same 68 training examples: 163,953 input-token and 147,597 supervised-target exposures. All 92 FP32 Adam states reached step 21 with zero skipped updates. Backbone/gate peak learning rates are 2e-5/1e-4, with warmup/cosine decay, an 8,192-input-token accumulation threshold, with whole-example overshoot and retained epoch tails, and loss normalized by the actual response-target count. There is no online teacher or KL objective.

Matched CPU continuation control

Two fresh CPU BF16 arms completed the same fixed 21-update schedule from the pre-SFT SHARED parent: original Fable responses versus geometry/mask-matched FineWeb-Edu replay. Each reused 68 examples for three epochs, 163,953 input tokens and 147,597 supervised targets. Lengths, prefix masks, EOS, update boundaries, optimizer settings and execution were matched. No updates were skipped, and final checkpoints were selected before PIQA scoring.

Experimental CPU child	PIQA correct / 1,838	Accuracy
Fable responses	1,072	58.32%
FineWeb replay	1,054	57.34%

The single predeclared primary comparison gives +18 answers / +0.979 percentage points, with 40 wins / 22 losses, exact two-sided paired p=0.0300. FineWeb finished +1 over the saved SHARED parent; the new Fable child finished +19. This supports a data-source difference in this matched run, without identifying which property of the examples caused it. It does not establish a formatting-only mechanism or broader reasoning gains. The experiment uses one training seed and previously evaluated public validation; this is not pristine confirmatory evidence. Replay is mask-matched, rather than conventional all-token continued pretraining. See the complete protocol, results and limitations.

These experimental children remain local; the released weights are unchanged. The released GPU-trained child's 1,070 count below remains its measured result.

Gates-only follow-up: qualification complete, training pending

The next controlled intervention freezes all 110,465,280 backbone parameters and every buffer, including router bias, and trains only the 1,771,776 gate parameters. Disposable CPU checks found exact checkpointed-versus-direct BF16 gate gradients and routing on two 128-token training prefixes, verified one gate-only optimizer update, and preserved the frozen state and gate-bypass function. A fresh-process resume rehearsal reproduced model, optimizer and RNG state exactly. These are implementation checks, not evidence of improved quality.

Full-length CPU profiling measured 14.07 seconds for 1×1,023 targets and 59.76 seconds for 8×1,023 targets, at approximately 2.83 and 11.07 GiB peak process RSS. The largest 12×1,023 group was skipped by the conservative RAM admission rule; it did not fail with an observed out-of-memory error. No retained gates-only training has run. The intended run retains the original 32 updates and 4,087,238 replay targets. See the CPU qualification and profiling report.

A subsequent disposable 5070 Ti Laptop GPU profile completed one original 127,600-target update with a batch cap of four in 421.21 seconds, peaking at 4.91 GiB allocated. This gives a bare 32-update extrapolation of 3.74 hours, before overhead and sustained-load effects. Larger batches were skipped by the conservative VRAM rule, not observed to OOM. No trained endpoint was retained.

On two 128-token TRAIN prefixes, default CUDA repeats changed 105/4,608 expert sets and produced different gate gradients. Under deterministic execution, checkpointed repeats and checkpointed/direct passes had bit-identical losses, all 18 gate gradients and all 108 routing calls. This is a sampled numerical control on native update-zero weights, not proof of the historical PIQA flip's cause. Deterministic full-update timing and CUDA resume remain untested. See the GPU timing and repeatability report.

Evaluation and the measured tradeoff

Full PIQA validation, CPU FP32

All 1,838 original validation items were scored once per saved arm under a frozen CPU FP32 protocol: mean complete-answer NLL, no EOS, capacity off, equal-length batches up to eight without padding and answer-only readout. This is a custom evaluation, not an exact reproduction of the earlier GPU BF16 counts. The previously reserved 1,338 rows are now consumed; no PIQA examples were used for new training or tuning.

Arm	Correct	Accuracy	Wilson 95% interval
Native, gates disabled	1065/1838	57.94%	55.67–60.18%
FULL@32	1062/1838	57.78%	55.51–60.02%
SHARED@32	1053/1838	57.29%	55.02–59.53%
Released response-SFT child	1070/1838	58.22%	55.95–60.45%

Native exceeds the majority-position control (928/1838) and all four recorded answer-length controls (870–968/1838). Its label-association Fisher p is 6.43e-12; the paired majority comparison gives 2.69e-6. This supports task-relevant predictive signal beyond those simple controls, without isolating physical reasoning or resolving pretraining contamination.

The child wins 37 and loses 20 cases versus SHARED: +0.925 percentage points, with raw paired two-sided p=0.0331 and Holm-adjusted p=0.198864 across the six exploratory pairwise comparisons. None of those six comparisons survives correction at familywise 0.05. Against native it wins 36 and loses 31 (raw p=0.6254), without an established improvement. Continued training and the response-data intervention remain confounded in the historical result. These results do not establish general instruction or reasoning gains, or a formatting-only mechanism. The new report contains the complete Holm table and the matched continuation design.

Eight batch-one checks per arm found no prediction changes and maximum mean-NLL difference 9.54e-7; that is a sample check, not a global bound. Most inputs (83.30% of candidates) fit in the first 50-token chunk, so this is a weak test of older-chunk KV sharing. Full results, paired comparisons and controls are in the controlled evaluation report.

General-language DEV and cache tradeoff

Matched evaluation scores 1,021,448 primary DEV targets across 923 components with capacity disabled. Four units / 3,442 targets overlapping earlier evaluated components are excluded. Absence of pretraining overlap is unestablished.

Saved arm, before response SFT	Primary CE, nats/target
Native, gates disabled	3.729070
FULL after adaptation	3.728203
SHARED after adaptation	3.785575

SHARED minus FULL is +0.057371 nat/target, paired component-bootstrap 95% interval [+0.051666, +0.063394]: about 5.9% higher perplexity. Bootstrap uses 2,000 draws, without training-seed uncertainty. FULL–native also changes gates and training exposure, preventing isolation of a MELT mechanism effect. The original 64-item PIQA diagnostic and its exploratory significance tests are documented in the controlled report. The full 1,838-item evaluation above is the primary PIQA result on this card.

The saved 32,065-target pilot also has update-zero scores: SHARED–FULL CE gap fell 0.115440 → 0.049794 nats over 32 updates (PPL penalty 12.24% → 5.11%). The larger cohort's 5.9% is a measurement after 32 updates, not an upper bound or a converged method result. Intermediate heldout scores were not saved, so convergence was not assessed. The cohorts cannot form one curve.

At batch one and 976 cached positions, an earlier pilot measured persistent BF16 KV storage of 13,492,224 bytes FULL versus 4,958,208 bytes SHARED: 63.251% less. In that pilot, inference allocator peaks fell only 0.891%. In the separate larger DEV workload, SHARED's peak allocated memory was 12.445% lower, while its measured forward time was 2.083% longer. These are different measurements; none is a 63% reduction in total GPU memory, and no speedup is demonstrated.

SFT validation response CE decreased 5.669114 → 5.327462 on five examples; final test response CE was 5.504149 on 12 examples. These tiny holdouts do not establish broad generalization. A historical pre-SFT GPU discrepancy on one near-tie PIQA item remains unresolved and is not attributed to SFT; the controlled report preserves the counts, margins and arithmetic diagnostics. The larger general-language DEV score was not rerun on the SFT child.

New CPU BF16 routing diagnostics also show that changing explicit accumulation order changes expert selections and candidate NLLs (maximum 0.0538 nat/token between matched ordered sums). Two identical default replays match exactly; all four tested conditions have identical prediction vectors. The original GPU routing traces were not saved: historical causation and any general ±1-item noise bound remain unestablished.

Four fixed prompts ran once on both parent and child with CPU FP32 greedy decoding, a 32-token cap and no rerolls:

1. “What does a mixture-of-experts layer do in a language model?”
2. “Write a Python function add(a, b) that returns their sum.”
3. “What is 17 + 25? Give the answer.”
4. “Explain in one sentence why Python list membership is usually slower than set membership for many items.”

All eight continuations reached the cap and failed their requests. The child repeats “the sum of the sum” for the coding prompt and writes 17 + 25 = 17. All eight unedited outputs and the full evaluation report accompany the release.

Running the model

This uses the bundled custom runtime. It is not currently compatible with AutoModelForCausalLM, the Transformers pipeline or hosted inference APIs. From the release repository root, install the pinned dependencies in a virtual environment, then use the supplied CLI:

Use Python 3.12. Install the Hugging Face CLI separately from the pinned model runtime dependencies:

python3.12 -m venv .venv-barbados
. .venv-barbados/bin/activate
pip install huggingface_hub
hf download Vykos/Barbados --local-dir Barbados
cd Barbados
pip install -r bundle/requirements.txt
CUDA_VISIBLE_DEVICES='' python -B bundle/run.py \
  --prompt "What is a mixture-of-experts layer?" --max-new-tokens 32

The CLI inserts the wrapper and emits JSON. Default inference is CPU FP32. Optional --device cuda requires the pinned Torch 2.13.0 CUDA 13.0 build and BF16 support; it uses BF16 parameter views with FP32 master state and routing. Run in a fresh process. Keep the bundle/ inventory intact: the runtime verifies files and hashes. Source, tokenizer and weights are bundled; training data, optimizer and cloud access are unnecessary. Relocated CPU execution matched the source reference without changing states; the public distribution also passed an independent relocated CPU check. These two-token checks verify delivery, not generation quality. Exact GPU training resume was not rehearsed.

Scope, provenance and licensing

Intended use is research on recurrence, sparse routing, differentiable KV sharing and small-budget training. Limitations include one seed, short context, limited training, tiny SFT splits, incomplete contamination coverage, unresolved PIQA disagreement and poor generations. Architectural superiority and useful capability are unestablished. Do not rely on its factual or coding advice.

The immutable bundle contains source attribution, training summaries, data receipts and hash-bound lineage. Its final packed state SHA-256 is 3df93e528bd1e821fc437b75aa3afb9bc6b129ffe7375b8192b915ddeef1f23d. Acknowledgements: the LOOM authors for the released backbone/training code; the Ouro and MELT authors for related recurrent-model research; Hugging Face contributors for FineWeb-Edu and the SmolLM tokenizer; and the public response dataset's uploader. These acknowledgements do not imply endorsement.

The uploader releases the original Barbados model weights and project-authored runtime under Apache-2.0; see LICENSE and NOTICE. Bundled upstream components retain their respective licenses and notices. Training datasets are not redistributed. FineWeb-Edu attribution and its ODC-By / CommonCrawl terms, the reused tokenizer's Apache-2.0 provenance and the Fable uploader's MIT declaration are documented in NOTICE. The Fable declaration does not authenticate teacher identity or establish rights in every response.

This is a separately packaged and licensed public distribution of the original trained artifact. Original manifest SHA-256: 6947858ef8baa2db08db06d80b58217ef246d66558af7819be1784954c41f64b. Public bundle manifest SHA-256: 7d5956fb54d733b0a419facf5c589f0a88c98cda2fbbaf40e8632d88d75a6151. The distribution adds comment-only change/license notices to seven source files and rebinds two runtime hash constants and their receipts. All 34 Python source ASTs are unchanged; all 19 other source files, model weights and tokenizer bytes are identical. Hash guards remain active; the inference algorithm is unchanged. The release audit maps every original and public payload hash and preserves the exact original manifests alongside it. These distribution edits do not retrain the model or change its measured results.