IMvision12/lamoe-247

LaMoE 247.65M

LaMoE is a 247,650,340-parameter hybrid language model developed by Gitesh (IMvision12). It was pretrained on 25.5 billion tokens using Google Colab and Modal, followed by supervised fine-tuning and identity correction. This release is the identity-corrected checkpoint step-00129185, used for the reported local benchmarks.

Architecture

• 16 decoder layers, hidden width 768, vocabulary 32,000 and context 2,048 tokens.
• Multi-head Latent Attention in every layer; Gated DeltaNet fusion in layers 4, 8, 12 and 16.
• Alternating dense SwiGLU and top-2-of-8 routed MoE, plus a shared expert.
• 134,404,132 active-path parameters under the documented counting convention.
• Tied embedding/output weights; original FP32 weights and frozen custom BPE tokenizer.

Load with the LaMoE source package

All Python implementation code is maintained in the GitHub source repository. This model repository holds weights, tokenizer/configuration files and documentation only. LaMoE is a custom architecture, so install its source package before loading it. trust_remote_code=True by itself cannot load this code-free model repository.

Download the source from GitHub, open the source folder, and install it in your own environment:

python -m pip install -e ".[transformers]"

Then register the installed implementation and load the model:

import torch
from lamoe.transformers import register
from transformers import AutoTokenizer, AutoModelForCausalLM

register()
repo = "IMvision12/lamoe-247"  # Or a local downloaded model-files folder.
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=False)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=False, dtype=torch.float32,
).eval()
inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Explain binary search."}],
    tokenize=True, add_generation_prompt=True, return_tensors="pt", return_dict=True,
)
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

The tested environment is Transformers 5.17.0 and PyTorch 2.11.0, using CPU. Pin the Hub revision and source version for a fixed model/code combination. The original trained weights and frozen tokenizer are unchanged. No retraining or identity/system prompt injection is performed by the adapter.

Supported behavior and boundaries

• Native user/assistant/system chat, standard dictionary-valued tool arguments, local loss calculation, full and incremental inference, greedy generation, beam search and save/load through Transformers.
• The cache retains compressed MLA history, GDN recurrent state and convolution history. Padded batches automatically use full-sequence generation without caching; left padding is the tokenizer default.
• Context remains 2,048 tokens. Inputs are never truncated; requested generation is capped to the remaining context. EOS and end-turn both stop generation.
• CPU uses reference GDN and SDPA internally. CUDA/FLA, quantization, distributed device maps, other Transformers versions, assisted/speculative decoding and static/offloaded caches are not certified by this export. Assisted decoding and foreign cache implementations are rejected explicitly.
• Intermediate hidden states, attention-map outputs, arbitrary position IDs, vocabulary resizing and inputs_embeds are not exposed by this wrapper.
• Literal native control-token strings inside message/tool content are rejected by the string-based chat template. Use the native token-level renderer for that special case. Tool definitions require an explicit initial system message. The template renders tool messages only; it never executes a tool.
• The original training source and original inference bundle remain separate. This model repository contains no Python implementation, datasets, collector code or optimizer/RNG states.

Evaluation

Historical local evaluations used an RTX 4060 laptop GPU, context 2,048, FP32 stored weights and BF16 FLA/SDPA inference. All local rows below are 0-shot. Counts, prompts and output caps vary by benchmark. These are separate metrics, not a combined average.

Benchmark	Metric	Score (%)
hellaswag	acc	29.7451
hellaswag	acc_norm	32.3541
piqa	acc	59.3036
piqa	acc_norm	60.0109
arc_challenge	acc	22.4403
arc_challenge	acc_norm	24.9147
winogrande	acc	50.9866
ifeval	inst_level_loose_acc	49.2806
ifeval	inst_level_strict_acc	46.8825
ifeval	prompt_level_loose_acc	34.0111
ifeval	prompt_level_strict_acc	32.5323
triviaqa	exact_match	1.0811
gsm8k	exact_match_flexible_extract	8.7945
gsm8k	exact_match_strict_match	0.0758
mmlu	acc	25.1460
humaneval_instruct	pass_at_1	15.2439
mbpp_instruct	pass_at_1	13.8000
bbh_cot_zeroshot	exact_match_flexible_extract	20.1198
bbh_cot_zeroshot	exact_match_strict_match	4.3158

GSM8K strict and flexible extraction are distinct metrics. IFEval prompt and instruction metrics use different denominators. MMLU is question-weighted across 57 subjects; BBH is question-weighted across 27 variants. MBPP includes its context-limit failure in the denominator. HumanEval and eligible MBPP candidates were graded twice in restricted networkless containers. See evaluation protocols, full CSV and published reference sources.

HellaSwag, PIQA, ARC-Challenge and WinoGrande

IFEval

GSM8K

MMLU

BBH

Published reference bars use differing prompts, shot counts, scoring and training regimes. They provide context, not a controlled ranking or a claim of general superiority.

Source: https://github.com/IMvision12/LaMoE