XERON-1.0-long

Long-context fine-tune of the XERON-1.0 agent (encoder + decision head), trained with the XERON RLCD-style objective on 8,192-token sequences.

XERON is a small language-model agent that emits a typed decision instead of free-form text: it scores a set of candidate actions (choice, no-ul, escalate, …) through a decision head on top of a ModernBERT encoder, and is trained with a group-relative reinforcement objective plus a language-modelling term.

Model details

Encoder	ModernBERT-style, hidden 1536 / 44 layers / 24 heads / inter 2304 / vocab 256,000
Attention	local window 128, every 3rd layer global
Parameters	1,329.0M (328 tensors)
Max sequence length	8,192 (head max 256)
Head layers	4
Dtype	bf16
Training dataset size	18,802 sequences
Effective batch size	32 (micro batch 8 × grad accum 4)
Optimizer	AdamW, enc lr 2e-5 / head lr 5e-5, weight decay 0.02
Schedule	1 epoch, 587 updates
RL weight / sigma	0.5, 0.3 → 0.2
Kitchen	1× A100 SXM4 80GB, 8,836.9 s wall (147 min), avg loss 1.4218

Batching used a length-bucketed, token-budget-aware planner (see the XERON repo, scripts/batch_sampler.py): batches are grouped by similar sequence length under a MAX_TOKENS_BATCH budget, which cut padding waste on this long-context subset to 1.76 % (vs ≈ 98 % for uniform 8,192-padding) and raised throughput.

Files

• model.safetensors — encoder + decision head weights (bf16)
• encoder/config.json — encoder configuration
• rl_agent_config.json — full training/agent configuration, including the training metadata block
• tokenizer/ — tokenizer files

Usage

import json, torch
from safetensors.torch import load_file

state = load_file("model.safetensors")
cfg = json.load(open("rl_agent_config.json"))
print(cfg["encoder"], cfg["head_layers"], cfg["max_len"])
# instantiate the XERON encoder+head from the PIXELZX/XERON repo, then:
# model.load_state_dict(state, strict=False)

The reference training and evaluation code lives in the XERON repository: https://github.com/PIXELZX0/XERON

Intended use & limits

• Research checkpoint for typed-decision agents; not safety-tuned and not instruction-tuned.
• Trained on a mixed English/Korean long-context decision corpus; behaviour outside that distribution is untested.
• Evaluation on public decision benchmarks is reported in the XERON repository.

Citation

@misc{xeron10long,
  title  = {XERON-1.0-long},
  author = {PIXELZX},
  year   = {2026},
  url    = {https://huggingface.co/PIXELZX/XERON-1.0-long}
}