ATH-MaaS/Lumen_wmt26

Lumen — WMT26 General MT Submission

This repository contains Lumen, a submission to the WMT 2026 General Machine Translation Shared Task.

Lumen is a 14.8B-parameter dense translation model built on the Qwen3-14B architecture. It covers all 23 official WMT26 translation directions, including several low-resource and regional varieties (Ligurian, Ladin, Northern Sámi, Egyptian Arabic).

At 14.8B parameters the model is eligible for the WMT26 constrained track, which requires open-weight models below 20B parameters.

Model details

Architecture	Qwen3ForCausalLM (dense, 40 layers, GQA with 40 Q / 8 KV heads)
Parameters	14.77 B
Hidden size / FFN	5120 / 17408
Vocabulary	152064 (Qwen2 BPE tokenizer)
Weights dtype	bfloat16, single model.safetensors (~29.5 GB)
Context length	4096 tokens (max_position_embeddings)
RoPE theta	10000

Supported language pairs

All 23 official WMT26 General MT directions:

Direction	Languages	Direction	Languages
en→zh	English → Chinese (Simplified)	en→is	English → Icelandic
en→zh-Hant-TW	English → Chinese (Traditional, Taiwan)	en→id	English → Indonesian
en→cs	English → Czech	en→kk	English → Kazakh
en→de	English → German	en→be	English → Belarusian
en→ru	English → Russian	en→hy	English → Eastern Armenian
en→uk	English → Ukrainian	en→arz	English → Egyptian Arabic
en→ja	English → Japanese	en→lij	English → Ligurian (Italy)
en→ko	English → Korean	en→lld	English → Ladin (Val Badia, Italy)
en→et	English → Estonian	en→se	English → Northern Sámi
en→th	English → Thai	cs→de	Czech → German
zh→ja	Chinese (Simplified) → Japanese	cs→uk	Czech → Ukrainian
		cs→vi	Czech → Vietnamese

Repository layout

.
├── model/                       # HF-format weights (upload target for Hugging Face)
│   ├── config.json
│   ├── generation_config.json
│   ├── chat_template.jinja
│   ├── special_tokens_map.json
│   ├── tokenizer.json
│   ├── tokenizer_config.json
│   └── model.safetensors        # 29,539,287,712 bytes, 443 tensors, bf16
├── inference/
│   └── run_eval_ruled_check.py  # submission decoding pipeline
├── requirements.txt
└── README.md

Prompt format

The model expects a Human: / Assistant: template (shipped as model/chat_template.jinja). A rendered prompt looks like:

Please translate the following text into {tgt_lang}:
#guidelines
{guidelines}
#src_text
{src_text}

The #guidelines block carries the per-domain instructions supplied with the WMT26 test sets (register, HTML preservation, handling of URLs and hashtags, and so on).

Usage

1. Dependencies

pip install -r requirements.txt

2. Minimal example (vLLM)

from transformers import AutoTokenizer
from vllm import LLM, SamplingParams

model_path = "model"  # local path, or the Hugging Face repo id once published

tokenizer = AutoTokenizer.from_pretrained(model_path)
llm = LLM(model=model_path, max_model_len=4096)

lang_name = {
    "zh": "chinese", "zh-tw": "traditional chinese", "cs": "czech", "de": "german",
    "uk": "ukraine", "vi": "vietnamese", "ja": "japanese", "ko": "korean",
    "ru": "russian", "th": "thai", "id": "indonesian", "et": "estonian",
    "is": "icelandic", "kk": "kazakh", "be": "belarusian", "hy": "armenian",
    "arz": "egyptian arabic", "lij": "ligurian", "lld": "ladin", "se": "northern sami",
}

source_text = "This paper presents the Lumen system, our submission to the WMT 2026."
target = "zh"

content = (
    f"Please translate the following text into {lang_name[target].title()}.\n#guidelines\n"
    f"#src_text\n{source_text}"
)
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": content}],
    tokenize=False,
    add_generation_prompt=True,
)

outputs = llm.generate(
    [prompt],
    SamplingParams(temperature=0.0, top_p=0.001, top_k=1, max_tokens=2048),
)
print(outputs[0].outputs[0].text.strip())

3. Reproducing the submission pipeline

inference/run_eval_ruled_check.py is the decoding pipeline used for the submission. It goes beyond plain greedy decoding:

1. Structure-aware splitting. Each source is classified as html, json or plain. HTML text nodes and translatable attributes (alt, title, placeholder, aria-label) and JSON string leaves are translated independently, then the original structure is rebuilt. Long plain text is split into sentence-aware chunks.
2. Rule-based quality checks. Every output is screened for empty results, length-ratio outliers, character-level decoder loops, repeated n-grams, insufficient target-script coverage, and source copying.
3. Anti-loop retries. Flagged samples are regenerated with repetition_penalty=1.3 and frequency_penalty=0.5 rather than a higher temperature, which was empirically ineffective against character-level runaway ("оооо…", "ііі…"). The better of the two candidates is kept.
4. Structural validation. Rebuilt HTML is compared against the source tag-count signature and JSON against the source shape, so structure-breaking outputs are rejected.

Input is a JSONL file where each line provides prompt, src, src_lang and tgt_lang:

python inference/run_eval_ruled_check.py \
    --model model \
    --input  testset/en-zh.jsonl \
    --output out/en-zh.jsonl \
    --max-tokens 2048 \
    --tensor-parallel-size 1

Two files are written: the main output with an added hypothesis field (plus _rule_flags, _retried, _kind diagnostics), and a <output>.failed.jsonl companion listing samples that were still flagged after retrying.

Notes and limitations

• Context length is 4096 tokens. The released config sets max_position_embeddings=4096 with rope_theta=10000, so this is a hard limit rather than a soft recommendation. Documents longer than this must be chunked; the pipeline above does so automatically. --max-tokens must leave room for the prompt inside this budget.
• Differences from stock Qwen3-14B. All layer dimensions match the official Qwen3-14B (5120 hidden, 40 layers, 40 Q / 8 KV heads, 17408 FFN, untied embeddings), but three config fields differ: vocab_size is 152064 rather than 151936, rope_theta is 10000 rather than 1000000, and max_position_embeddings is 4096 rather than 40960. The tokenizer holds 151669 real tokens, so the vocabulary difference is padding only.
• Compatibility. config.json follows the transformers 5.x schema, using dtype and a nested rope_parameters block. transformers 4.x does not recognise these keys, so pass the dtype explicitly (dtype=torch.bfloat16) to avoid loading the model in fp32.
• Target-script check on JSON. The script-coverage rule measures the ratio of target-script characters over the whole output. For JSON payloads the untranslated ASCII keys dilute this ratio, so short dictionaries can be flagged even when the translation is correct. These land in .failed.jsonl for review rather than being discarded.

License

Released under the Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0.txt, SPDX-License-Identifier: Apache-2.0).