FAIRC/mowe-moe-768-19l

Standard MoE 768, 19 Layers

Inference checkpoint for the standard mixture-of-experts baseline released with Mixture of Wide Experts. Every selected expert reads and writes the full model width.

Property	Value
Parameters	363.71M
Transformer layers	19
Model width	768
Attention heads	12
Experts	768
Hidden units per expert	8
Selected experts per token	38
Context length	1,024
Tokenizer	GPT-2

The checkpoint is an inference-only safetensors state dictionary. It contains no optimizer state. This is a base language model and is not instruction-tuned.

Loading

Install the companion MoWE extension for OpenLanguageModel, then load this repository directory:

import torch

from mowe import load_checkpoint

model = load_checkpoint(
    "/path/to/moe-768-19l",
    device="cuda",
    dtype=torch.bfloat16,
).eval()

Limitations

This is a small research model. It may produce incorrect, biased, unsafe, or memorized text and should not be used as a production assistant without separate evaluation and safeguards.

License

Released under the Apache License 2.0. Please cite the accompanying MoWE paper.