Title: UHGEval: Benchmarking the Hallucination of Chinese LLMs via Unconstrained Generation
Benchmark: UHGEval
Summary: UHGEval introduces a 5k samples benchmark for evaluating hallucination in Chinese large‑language models.  The authors collect 2015‑2017 Chinese news articles, ask five different Chinese LLMs to continue each “news beginning” without any restrictive prompts, then automatically rank, label (keyword‑level), and human‑verify hallucinations. The paper also ships a modular evaluation framework supporting three task forms: discriminative, selective, and generative.
Contribution: - The paper presents the first large-scale unconstrained hallucination benchmark for Chinese LLMs, addressing a major gap in current evaluations that rely on constrained generation techniques (e.g., directed prompts or perturbations). This enables more realistic benchmarking of model behavior in real-world settings.
- It introduces a hybrid labelling pipeline combining automatic keyword-level annotation via GPT-4 and human re-verification, ensuring scalable yet accurate hallucination detection which more fine-grained than typical sentence/document-level annotation.
- The evaluation framework is notably broad, supporting three evaluation forms including: discriminative (detecting hallucinations), selective (choosing hallucination-free outputs), and generative (continuation from prompt), which allows multi-angle assessment of model robustness.
- The benchmark is used to empirically evaluate 11 major LLMs (including 8 Chinese LLMs and 3 GPT models), revealing useful trends (e.g., GPT’s strong discriminative ability but weaker Chinese generative performance), and highlighting the "seesaw" effect between task types.
- Overall, UHGEval sets a new standard for hallucination evaluation in low-resource languages (Chinese), with a modular, extensible toolkit that could be generalized to other languages and domains.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Hallucination / factual consistency in generation
Phenomenon defined by authors: Yes
Phenomenon definition: Hallucination occurs when LLMs produce content that is factually incorrect or unsupported by the source or real-world knowledge, especially in unrestricted, spontaneous generation settings.
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Purpose notes: Focuses on unconstrained hallucinations; contrasts with prior constrained‑prompt datasets
Task definition: (i) Given a continuation, decide if it contains hallucinations (discriminative); (ii) pick the hallucination‑free option from a pair (selective); or (iii) generate a continuation that avoids hallucination, later scored by reference metrics (generative).
Task item definition: One row contains: article ID, headline, date, type (DOC/KNO/NUM/GEN), newsBeginning, LLM‑generated hallucinatedContinuation, per‑keyword labels (reasonable / unreasonable), real continuation, and remaining article text.
Task source: Real task examples (e.g. GitHub issues), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: News from major Chinese outlets (Jan 2015 – Jan 2017); five Chinese LLMs (ChatGLM2‑6B, Baichuan2‑13B, Qwen‑14B, InternLM‑20B, Xinyu‑7B) produce continuations; automatic ranking + GPT‑4 keyword labeling + human re‑check.
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Task ecology detail: Models must handle real news beginnings but hallucinations are induced by LLM continuations rather than reporters.
Dataset size: 5,141
Dataset metadata available: Yes
Dataset metadata detail: news category (DOC/NUM/KNO/GEN), generation LLM, lengths, keyword counts
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Response format detail: Discriminative/Selective expect “1/0” or chosen option; Generative expects unconstrained Chinese text.
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), BertScore, kwPrec
Metric aggregation: Simple Mean
Metric statistics: Mean
Metric subscores: Yes
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Authors describe automatic‑plus‑manual labelling pipeline, double‑checked subsets, and identify remaining noise as limitation.
Taxonomy: Language Modelling / Hallucination
New taxonomy: General Capability / Hallucination
Other notes: The paper acknowledges in Appendix G that there is a data skew due to an imbalance in the number of hallucinated continuations generated by the five LLMs, and it highlights this as an area for future work.