Title: CARE-MI: Chinese Benchmark for Misinformation Evaluation in Maternity and Infant Care
Benchmark: CARE-MI
Summary: CARE‑MI introduces a 1,612‑item Chinese benchmark that tests large‑language‑model misinformation in long‑form answers on the sensitive domain of maternity and infant care. Items are derived from biomedical KGs and medical‑licensing MCQ banks, converted mostly with LLM + rule pipelines into true/false and open‑ended questions, paired with retrieved evidence, and vetted by medical experts. The authors evaluate several Chinese LLMs, provide a human baseline, and release a fine‑tuned LLaMA‑13B “judge” model to automate scoring.
Contribution: - First Chinese, expert‑checked dataset for domain‑specific misinformation in LF generation.
- Transferable data‑construction pipeline (true/false + OE Q generation, knowledge retrieval, expert vetting).
- Off‑the‑shelf judgment models showing high Pearson ρ (0.87–0.90) with human scores, reducing eval cost.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Factual correctness & explanation quality (misinformation detection)
Phenomenon defined by authors: Yes
Phenomenon definition: The risk of misinformation, stemming from the generation of erroneous, deceptive, irrational, or substandard information, defined as LLM outputting false, misleading, nonsensical or poor quality information, without malicious intent of the users.
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Purpose notes: Focuses on high‑risk healthcare advice; highlights long-form generation failures.
Task definition: Given a maternity/infant‑care question (T/F or open‑ended) plus retrieved evidence, generate an answer; evaluation judges factual correctness and interpretability.
Task item definition: A single row consists of {question, answer placeholder, evidence paragraphs, expert labels}.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: KG triples → rule sentences; MCQ → GPT‑3.5 & ChatYuan QA2D + negation/replacement; questions generated with ChatYuan.
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: Synthetic but mirrors real consumer health Q&A.
Dataset size: 1,612 (Test)
Dataset metadata available: Yes
Dataset metadata detail: source (BIOS / CPubMed / MLEC‑QA / MEDQA), question type (TF/OE), length stats.
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Simple mean
Metric subscores: Yes
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Provides expert agreement stats, compares human vs LLM, ablates judgment model with/without evidence (ρ↑), and discusses linear relation between correctness & interpretability, limitations
Taxonomy: Medicine / 
New taxonomy: Specific Application / Medicine