OpenTWBench

An open evaluation suite for Taiwan-domain knowledge in large language models.

🔗 opentwbench.ai — leaderboard, subject map, and how the data is built

Most Chinese-language benchmarks are built from mainland sources, in Simplified Chinese, about mainland institutions. A model can score well on them while knowing nothing about the Labor Standards Act, the Household Registration Act, or how a Taiwanese pharmacist is licensed. These datasets exist to make that gap measurable.

The questions come from Taiwan's national examinations, published as open data by the Ministry of Examination (考選部) — professionally written, officially answer-keyed, and covering essentially every regulated profession in the country.

The datasets

One dataset per 學科 (academic subject), named tw-<subject>-bench, in the Twinkle Eval MCQ format (question, A–D, answer) with provenance columns alongside. Subjects are the unit, not the profession sitting the exam: 神經疾病物理治療學 and 骨科疾病物理治療學 are separate benchmarks, because a model can know one and not the other.

from datasets import load_dataset
ds = load_dataset("OpenTWBench/tw-medicine-bench", split="test")

Reading the scores honestly

• These are public past papers, very likely in pretraining corpora. Use the suite for relative comparison between models, not to claim a model "passes" an exam.
• Check the parse rate before believing a low score — a model that knows the answer but writes it in an unexpected shape is a formatting failure.
• Shuffle the options. The source papers carry an answer-position skew.

Taiwan law is covered separately and in depth by lianghsun/tw-legal-benchmark-v2; questions already published there are excluded from this org, item by item.

Contributing

A wrong answer key is worse than a missing question. Open a discussion on the dataset it affects, quoting the paper_id and q_no, and it can be traced to the original PDF.

Questions and answer keys are official publications of 考選部, released as open data. Packaging and metadata: Apache-2.0.