E6E831728/fem-multi-mesh-1p7b

FEM-Inspired Causal Multi-Mesh LM

An experimental attention-free causal language model using:

• frozen injective Binary16 token coordinates;
• deterministic glyph/form observations;
• lossless short-span Structured Binary Tiles;
• a causal coarse history pyramid;
• latent scratch cells;
• explicit restriction and prolongation;
• shared iterative ConvGLU solvers.

This is a base model, not an instruction-tuned assistant.

Important interpretation

This release demonstrates that a heterogeneous causal multi-mesh system can learn coherent autoregressive text without attention.

It does not establish:

• literal finite-element equivalence for language;
• superiority over Transformers;
• better perplexity or throughput;
• autonomous reasoning;
• elimination of autoregression.

The implementation is best described as:

an FEM-inspired attention-free causal multi-mesh operator network.

Training checkpoint

Field	Value
Optimizer step	109200
Processed prediction targets	28626124800
Stored tensor values	1,726,969,344
Approximate parameter values	1,675,064,832
Fixed-buffer values	51,904,512
Weight SHA-256	935b2665ecc9d365231ae798916d44eb9477bdf16a5e9ec74f0374d8e48f4c12

The source directory name may contain 100b, but the authoritative training amount is the tokens_seen value above.

Architecture

The same text is represented as:

Binary16 fine token field
Structured Binary Tiles
glyph/form field
causal coarse hierarchy
latent scratch field

A completed coarse block may affect only later token positions.

Loading

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

path = "E6E831728/fem-multi-mesh-1p7b"

tokenizer = AutoTokenizer.from_pretrained(path)

model = AutoModelForCausalLM.from_pretrained(
    path,
    trust_remote_code=True,
    dtype=torch.bfloat16,
).to("cuda").eval()

inputs = tokenizer(
    "Gravity is",
    return_tensors="pt",
    add_special_tokens=False,
).to("cuda")

with torch.inference_mode():
    logits = model(
        input_ids=inputs["input_ids"],
        attention_mask=inputs.get(
            "attention_mask"
        ),
    ).logits

print(logits.shape)

Reference generation is available through:

generated = model.generate_simple(
    input_ids=inputs["input_ids"],
    max_new_tokens=9,
    temperature=0.0,
    eos_token_id=tokenizer.eos_token_id,
)

generated_text = tokenizer.decode(generated[0], skip_special_tokens=True)

print(generated_text)

Runtime limitations

• No attention KV cache is implemented.
• Generation recomputes the active context.
• Batched padded inference is not supported reliably; use batch size 1 for evaluation.
• The trained context is defined by config.block_size.
• The model's training loss API uses externally shifted labels. Standard evaluation should consume logits rather than rely on the training loss.
• The frozen form-codebook is stored in model.safetensors; it is not rebuilt during loading.

Intended use

Research on:

• attention-free language modeling;
• fixed token coordinates;
• heterogeneous text discretizations;
• explicit causal transfer operators;
• shared iterative computation.

Do not use for high-stakes decisions.

Audited standard language-model evaluation

Benchmark metric	Audited result
HellaSwag acc	29.00 ± 0.45
HellaSwag acc_norm	30.77 ± 0.46
ARC-Easy acc	53.37 ± 1.02
ARC-Easy acc_norm	47.14 ± 1.02
ARC-Challenge acc	21.59 ± 1.20
ARC-Challenge acc_norm	23.55 ± 1.24
PIQA acc	62.95 ± 1.13
PIQA acc_norm	62.62 ± 1.13
WinoGrande acc	49.33 ± 1.41
OpenBookQA acc	19.60 ± 1.78
OpenBookQA acc_norm	31.20 ± 2.07
CommonsenseQA acc	18.59 ± 1.11
MMLU 0-shot	23.95 ± 0.36
MMLU 5-shot	25.05 ± 0.36
LAMBADA accuracy	4.87 ± 0.30
LAMBADA perplexity	1707.25 ± 82.56
WikiText word perplexity	65.02
WikiText byte perplexity	2.18
WikiText bits/byte	1.13

This is evaluation of the standalone causal Multi-Mesh LM. No external document elements or VM calls are enabled. These scores do not establish superiority to a Transformer.

The model uses FEM-inspired multi-field operators; it does not implement or solve a physical finite-element PDE.