Introducing Terminal-Bench-LILT: Multilingual Agentic Benchmark Grounded in Language, Region, and Culture

63.1%. That's the best resolution rate any frontier model achieved on Terminal-Bench-LILT, our new multilingual coding-agent benchmark.

300 tasks across Arabic, Czech, German, Spanish, Hindi, Japanese, Korean, Serbian, Turkish, and Chinese. 30 per language, authored by native-speaking engineers rather than translated from English. Each task ships with instructions in both the native language and English, a Docker environment with native-language data assets, an oracle solution, and deterministic tests. Runs under Harbor, following the original Terminal-Bench task structure.

Seven models evaluated with the fixed Terminus-2 agent, 5 runs per task-model pair: GPT-5.5 (63.1), Gemini 3.5 Flash (61.2), Claude Opus 4.8 (60.2), Muse Spark 1.1 (60.2), Claude Opus 4.7 (56.8), Gemini 3.1 Pro (55.9), GPT-5.4 (31.2). Difficulty is calibrated from observed pass rates: 133 easy, 86 medium, 81 hard. A subset was solved by no model.

Two things worth flagging. First, replacing native-language instructions with English translations (German, Hindi, Chinese) changed pass rates by at most 7pp, so instruction comprehension is not the binding constraint. Second, the benchmark's largest categories are "Cultural and Other" (129 tasks: lunar and Hijri calendar conversion, social-system rules, colloquial conventions) and text conversion (76), ahead of classical internationalization (52). The coverage is deliberately weighted toward implicit locale and cultural knowledge rather than library-level i18n.

Quality control is four layers: deterministic structure and oracle checks, LLM judges for native/English instruction alignment and categorization, then two rounds of human review for realism, clarity, and overlap.

Read the full technical report here: https://arxiv.org/pdf/2608.28641

Sample dataset here: https://github.com/lilt/terminal-bench-lilt