FL-12B-MoE-2

A COBOL-specialised Mixture-of-Experts model, adapted from JetBrains/Mellum2-12B-A2.5B-Thinking.

12B total parameters, 2.5B active per token. Built for writing and repairing fixed-format COBOL that compiles under GnuCOBOL 3.2.

Results

Evaluated on COBOLEval: 146 problems translated from HumanEval into COBOL, 821 test cases, scored by compiling and executing every generated program with GnuCOBOL 3.2. A solution counts only if it passes every test for its problem.

	pass@1	compile rate	test pass rate
Base model (Mellum2-12B-A2.5B-Thinking)	0.68%	12.18%	3.05%
FL-12B-MoE-2	13.70%	48.36%	23.51%
	20x	4.0x	7.7x

Specialisation takes the base model from 1 solved problem out of 146 to 20, and from 100 compiling test programs to 397.

For reference, the COBOLEval authors report 10.27% pass@1 and 47.94% compile rate for GPT-4 on the same benchmark.

All figures use identical decoding: greedy (temperature 0), one sample per problem, 2048 maximum new tokens, 8192 context, chat prompt format. The same column-normalisation post-processing is applied to every model measured, including the base.

What the model is good at

Writing fixed-format COBOL that compiles. Column discipline is respected: sequence area in columns 1 to 6, indicator in column 7, code from column 8.

Callable subprograms. Recognises a LINKAGE SECTION with PROCEDURE DIVISION USING and returns control with GOBACK rather than STOP RUN, which would terminate the entire run unit and leave the caller without control.

Reading existing COBOL. Record layouts, byte counts, control flow, and "what does this program do" questions, from roughly 24M tokens of COBOL comprehension data in the training mix.

Repairing compiler errors. Trained on paired defect and fix examples covering column violations, missing FUNCTION keywords, DIVIDE syntax, INSPECT operands, invented intrinsics, figurative constants and subprogram termination.

Intended use

Assisting developers working with COBOL: drafting subprograms, completing data divisions, diagnosing compiler errors, and explaining legacy code. Generated code should be compiled, reviewed and tested before use. Not suitable for unsupervised modification of production mainframe systems.

Training

Adapted with LoRA on the attention projections only.

Base	JetBrains/Mellum2-12B-A2.5B-Thinking
Architecture	MoE, 64 experts, 8 active per token, 28 layers, d_model 2304
Attention	GQA 32Q/4KV, sliding window 1024, 131,072 context
Adapter	LoRA r=32, alpha 64, dropout 0, on q/k/v/o_proj (112 modules)
Trainable	16,515,072 parameters, 0.136% of the model
Never adapted	MoE router, expert MLPs, lm_head, embeddings
Precision	bfloat16, no quantisation during training
Sequence	8192 with packing, loss on assistant turns only
Optimiser	AdamW 8-bit, cosine schedule, 3% warmup, weight decay 0.01
Total tokens	187M
Total steps	1,428
Hardware	1x RTX 5090 (32 GB), roughly 22 GPU-hours

The MoE router is deliberately left untouched. Adapting it changes which experts fire and degrades the model in ways that surface as diffuse quality loss rather than as an error.

Data

The bulk of the corpus is COBOL, around 70% of tokens, with the remainder covering JCL, C#, x86 assembly and step-by-step reasoning traces. Tasks include source continuation, full subprogram generation, and instruction data on reading and analysing existing COBOL.

On top of that sit small, targeted sets built by mining the same corpus for constructs the model was getting wrong: subprogram termination, column discipline, the FUNCTION keyword, DIVIDE ... GIVING, INSPECT operands, reference modification X(pos:len) in place of a non-existent SUBSTRING, and figurative constants. Repair examples were produced by injecting a known defect into working code, so the reference answer is the original source rather than a generated one.

The list of valid intrinsic functions was derived by counting FUNCTION occurrences across 4,293 real programs rather than written from memory.

Usage

GGUF quantisations are provided for llama.cpp and compatible runtimes.

Quantisation	Notes
Q8_0	Highest fidelity
Q6_K	Close to Q8_0, smaller
Q4_K_M	Recommended balance
Q2_K	Smallest, expect noticeable quality loss

For a Mixture-of-Experts model with 2.5B active parameters, quantisation error concentrates in a small number of active experts, so Q2_K degrades more sharply than it would on a dense model of similar size. Q4_K_M is the practical floor.

llama-cli -hf DarkKnighToS223/FL-12B-MoE-2:Q4_K_M

llama-server -hf DarkKnighToS223/FL-12B-MoE-2:Q4_K_M --port 8080

Recommended sampling: temperature 0 for code generation, or 0.3 with top_p 0.9 when you want variety.

License and attribution

Released under CC-BY-NC-4.0.

Derived from JetBrains/Mellum2-12B-A2.5B-Thinking, licensed under Apache 2.0. The Apache 2.0 notice and attribution for the base model are retained. The non-commercial condition applies to this derivative work and to the adaptation contributed here. If you intend to use this model commercially, review the base model's terms and seek your own legal advice.

Training data was drawn from public COBOL and mainframe source corpora.

Citation

@misc{fl12bmoe2,
  title  = {FL-12B-MoE-2: a COBOL-specialised Mixture-of-Experts model},
  author = {DarkKnighToS223},
  year   = {2026},
  url    = {https://huggingface.co/DarkKnighToS223/FL-12B-MoE-2}
}