SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tong Xiang\*, Liangzhi Li\*, Wangyue Li\*, Mingbai Bai\*, Lu Wei\*, Bowen Wang\*, Noa Garcia\*
$\spadesuit$  Meetyou AI Lab,  $\text{心}$  Xiamen Key Laboratory of Women's Internet Health Management,  $\spadesuit$  Southwest University of Finance and Economics,  $\clubsuit$  Osaka University {xiangtong, liliangzhi, liwangyue, baimingbai, weilu}@xiaoyouzi.com, {wang, noagarcia} @ids.osaka-u.ac.jp
The recent advances in natural language processing (NLP), have led to a new trend of applying large language models (LLMs) to real-world scenarios. While the latest LLMs are astonishingly fluent when interacting with humans, they suffer from the misinformation problem by unintentionally generating factually false statements. This can lead to harmful consequences, especially when produced within sensitive contexts, such as healthcare. Yet few previous works have focused on evaluating misinformation in the long-form (LF) generation of LLMs, especially for knowledge-intensive topics. Moreover, although LLMs have been shown to perform well in different languages, misinformation evaluation has been mostly conducted in English. To this end, we present a benchmark, CARE-MI, for evaluating LLM misinformation in: 1) a sensitive topic, specifically the maternity and infant care domain; and 2) a language other than English, namely Chinese. Most importantly, we provide an innovative paradigm for building LF generation evaluation benchmarks that can be transferred to other knowledge-intensive domains and low-resourced languages. Our proposed benchmark fills the gap between the extensive usage of LLMs and the lack of datasets for assessing the misinformation generated by these models. It contains 1,612 expert-checked questions, accompanied with human-selected references. Using our benchmark, we conduct extensive experiments and found that current Chinese LLMs are far from perfect in the topic of maternity and infant care. In an effort to minimize the reliance on human resources for performance evaluation, we offer off-the-shelf judgment models for automatically assessing the LF output of LLMs given benchmark questions. Moreover, we compare potential solutions for LF generation evaluation and provide insights for building better automated metrics. Code and models are available at https://github.com/Meetyou-AI-Lab/CARE-MI.
Over the past few years, the community witnesses the rise of pretrained autoregressive large language models (LLMs) [Brown et al., 2020a, Lieber et al., 2021, Rae et al., 2021, Smith et al., 2022]. These mega models have shown transcendent power [Singhal et al., 2022], achieving and even surpassing human performance in a breadth of tasks [Clark et al., 2019, Rajpurkar et al., 2018, 2016, Wang et al., 2019], and cover various possible application scenarios such as code generation [Bommasani et al., 2021], healthcare chatbots [Nuance, 2023] and keyword matching [OpenAI, 2022].
As the size of LLMs continues to grow, the potential harms caused by model-generated text are of increasing concern [Bender et al., 2021, Bommasani et al., 2021, Dinan et al., 2021, Kenton et al., 2021]. A comprehensive taxonomy of ethical risks associated with recent LLMs is introduced in [Weidinger et al., 2022]. One of its most pressing concerns is the risk of misinformation, stemming from the generation of erroneous, deceptive, irrational, or substandard information, defined as LLM outputting false, misleading, nonsensical or poor quality information, without malicious intent of the users. Potential harms of misinformation range from deceiving a person, causing physical damage, and amplifying the society's distrust on the usage of LLM-incorporated systems [Weidinger et al., 2022]. For example, Galactica [Taylor et al., 2022], an LLM trained using a wide range of scientific sources including papers, reference materials, knowledge bases, etc., was reported to generate a fake study about the benefits of eating crushed glass. The aforementioned instance highlights the limitations of current LLMs in consistently delivering factually correct responses to user queries, a deficiency that potentially yield hazardous outcomes.
Despite the existence of several prior analyses, concerns are still far from being fully resolved. First of all, the evaluation of misinformation harm in existing LLMs has been conducted only using relatively simple formats such as multiple-choice (MC) question-answering (QA) task [Hendrycks et al., 2021a] or cloze task [Petroni et al., 2019a]. These tasks are usually either formalized as a completion task, where LLMs are required to predict only a single token instead of generating full sentences, or using relatively simple evaluation metrics, e.g., accuracy, perplexity [Dinan et al., 2019], ROUGE score [Lin, 2004] and BLEU score [Khashabi et al., 2021a], for the ease of evaluation [Hendrycks et al., 2021b]. In the context of long-form  $(\mathrm{LF})^4$  generation, Zhang et al. [2023] found that an incorrectly generated token at the outset will often be followed by a subsequent erroneous explanation; more generally, LLMs are reported to have the tendency to generate false statements, ranging from subtle inaccuracies to blatantly inauthentic claims [Lin et al., 2022]. These phenomena highlight the importance of conducting LF generation evaluations for LLMs. Yet, there are no sufficient datasets available for such evaluations, especially in knowledge-intensive domains. Furthermore, due to the unbalanced distribution in available language-related resources [Zeng et al., 2023], a substantial number of existing datasets and benchmarks only focus on measuring the misinformation in English, which impedes similar evaluations from being performed in other languages. A brief summary of previous datasets on misinformation evaluation is shown in Table 1; even though some of them are not initially designed for misinformation evaluation, we found that they can be easily used for it.
To motivate the misinformation evaluation on LLMs, we propose Chinese benchmark for misinformation Evaluation in Maternity and Infant care (CARE-MI), a benchmark to test medical-related misinformation of LLMs in Chinese. CARE-MI specifically focuses on the sub-domain of maternity and infant care, a topic in which users are prone to generate questions, especially expectant and first-time parents, about multiple issues, e.g., pregnancy and/or baby-related illnesses, symptoms, milestones, etc., and in which LLMs should respond factually and without errors. Although there are already existing datasets focusing on the medical domain, until now, there is no benchmark suitable for evaluating misinformation on such an important and sensitive topic as the maternity and infant care, neither in English nor in Chinese. The closest dataset to CARE-MI is MATINF [Xu et al., 2020a], which, differently from us, focuses on community QA and contains neither expert-level annotations nor supporting evidence documents, making it not suitable for misinformation evaluation. Our benchmark, however, is not designed for directly evaluating user-LLM interactions, as most of our questions require expert-level knowledge and contain medical norms, but it is a necessary prerequisite for LLMs to perform well in those cases.
Additionally, we conduct an extensive evaluation on recent Chinese LLMs. The results indicate that all current models are not able to maintain an acceptable performance while answering domain-related questions. We further explore the paradigm for automatic evaluation on LF generation of LLMs
2This undesired phenomenon of LLMs sometimes is called hallucination by some previous work [Ji et al., 2023a], a term that is initially mentioned in the context of psychology, defined as "percept, experienced by a waking individual, in the absence of an appropriate stimulus from the extracorporeal world" by Blom [2010]. However, we stick to the description of misinformation throughout our paper for consistency.
<sup>3</sup>https://news.yahoo.com/meta-ai-bot-contributed-fake-212500291.html
4LF generation is a widely used concept [Fan et al., 2019], referred as generation of text that "span multiple sentences" or "paragraph-length". We found no stringent standard that can be utilized to distinguish LF generation against shorter ones; however, we put nearly no limitation on models' generation, i.e., we let the models generate sentences at will until maximum length is reached. See Appendix B.6 for more details.
Table 1: A brief summary of datasets in misinformation evaluation.  $LF$ : whether the dataset is designed for LF generation task; Supervised: whether the dataset construction involves human supervision.  $\alpha$ We refer to the selected samples that have been annotated by annotators.  $\beta$ We are only referring to the expert annotated part of PubMedQA.  $\gamma$ For Chinese mentioned here, we are referring to both traditional and simplified Chinese.
Dataset	Language	#Question	LF	Supervised
Multiple
COMMONSENSEQA [Talmor et al., 2019]	English	12,247	✘	✓
Wizard of Wikipedia [Dinan et al., 2019]	English	201,999	✓	✘
LAMA [Petroni et al., 2019b]	English	-	✘	✘
NQ [Kwiatkowski et al., 2019]	English	323,045	✓	✓
ELI5 [Fan et al., 2019]	English	≈272,000	✓	✘
MMLU [Hendrycks et al., 2021a]	English	15,908	✘	✘
COM2SENSE [Singh et al., 2021]	English	≈4,000	✘	✓
GOOQ [Khashabi et al., 2021b]	English	≈3,100,000	✓	✘
KMIR [Gao et al., 2022]	English	16,000α	✘	✓
TruthfuQA [Lin et al., 2022]	English	817	✓	✓
ScienceQA [Lu et al., 2022]	English	21,208	✘	✓
M3KE [Liu et al., 2023]	Chinese	20,477	✘	✘
Legal
JEC-QA [Zhong et al., 2019]	Chinese	26,365	✘	✘
CaseHOLD [Zheng et al., 2021]	English	53,137	✘	✘
Medical
cMedQA2 [Zhang et al., 2018]	Chinese	108,000	✓	✘
PubMedQA [Jin et al., 2019]	English	≈1,000β	✘	✓
MATINF [Xu et al., 2020a]	Chinese	≈1,070,000	✘	✘
MEDQA [Jin et al., 2020]	Chinese/Englishγ	61,097	✘	✓
CMeIE [Guan et al., 2020]	Chinese	22,406	✘	✓
BioLAMA [Sung et al., 2021]	English	≈49,000	✘	✘
MLEC-QA [Li et al., 2021a]	Chinese	136,236	✘	✓
CARE-MI (ours)	Chinese	1,612	✓	✓
as a completion to the benchmark dataset for the purpose of efficient and accurate evaluation. We test multiple judgment models trained on the same set of questions as in the benchmark, with 1) synthetically generated positive and negative answers, and 2) expert-level annotations on the models' outputs, and 3) expert-annotated knowledge. The whole pipeline, encompassing the development of the benchmark and the training of the judgment model, can also serve as an innovative paradigm for creating similar benchmarks in other knowledge-intensive domains or low-resourced languages.
2 CARE-MI data acquisition
With no existing datasets on the topic (maternity and infant care), language (Chinese), and task (misinformation evaluation in LF generation) of interest, we construct CARE-MI from multiple data sources. We utilize two knowledge graph (KG) datasets and two MCQA datasets as our data sources. The collected data is then filtered to align with our focused topic.
KG datasets We rely on two medical-based KG datasets in Chinese: Bio-Medical Informatics Ontology System (BIOS) [Yu et al., 2022] and CPubMed [CPu, 2021], both of which come with <head, relation, tail> triplets. BIOS is a machine-generated bio-medical KG built on top of the abstracts and central articles from PubMed, which is a search engine for bio-medical articles. Both head and tail in BIOS are concepts representing the nodes in the KG; each concept includes multiple terms that are considered synonymous. In total, it contains 4.1 million concepts, 7.4 million terms, and 7.3 million relations. As the data is gathered from PubMed, BIOS triplets are collected in English and later translated into Chinese [Luo et al., 2021]; translation quality is ensured by applying back-translation and filtering out samples with low confidence. On the other hand, CPubMed is an
<sup>5</sup>https://pubmed.ncbi.nlm.nih.gov/
Table 2: QA pair examples in MCQA datasets.  $Q_{\cdot}$  stands for question and  $A$  for answer. The correct answers are underlined. English translations are shown for reference.
Source	Example
MLEC-QA	Q.	男婴,2个月,生后20天开始出现呕吐,进行性加重,有时呈喷射性,多发性于喂奶后半小时之内,呕吐物多为奶凝块,不含胆汁,吐后食欲极好,但体重不增。考虑的诊断是( )A 2-month-old male infant, who started experiencing vomiting 20 days after birth.The vomiting progressively worsened, sometimes with projectile vomiting, mostly occurring within half an hour after breastfeeding. The vomit often contains milk curds but not bile. After vomiting, the baby has an excellent appetite, but his weight does not increase. The considered diagnosis is ( )
A.	A.先天性肥厚性幽门狭窄B.先天性肠扭转不良C.胃食管反流病D.十二指肠溃疡E.肠套叠	A. Congenital hypertrophic pyloric stenosisB. Congenital intestinal malrotationC. Gastroesophageal reflux diseaseD. Duodenal ulcerE. Intussusception
MEDQA	Q.	唯一能通过胎盘进入胎儿体内的免疫球蛋白是( )The only immunoglobulin that can pass through the placenta and enter the fetus is ( )
A.	A. IgM B. IgG C. IgA D. IgE
open-source KG dataset constructed by the full-text Chinese periodical data of the Chinese Medical Association. It contains more than 1.7 million entities and around 4.4 millions of triplets. Examples of BIOS and CPubMed triplets can be found in Appendix B.2.
MCQA datasets Bio-medical KGs inherently entail factual information about the medical domain, yet the information is limited due to the intrinsic format of triadic representation. To collect more complex samples, we further include two Chinese bio-medical MCQA datasets: MLEC-QA [Li et al., 2021b] and MEDQA [Jin et al., 2020]. Both datasets are collected from the National Medical Licensing Examination in China (NMLEC), which is a Chinese official exam to evaluate professional knowledge and skills for medical and clinical practitioners [Li et al., 2021b]. Samples from these datasets can be categorized into several bio-medical sub-domains. Specifically, MLEC-QA covers Clinic, Stomatology, Public Health, Traditional Chinese Medicine, and Chinese Western Medicine domains, while MEDQA only covers the sub-field of Clinic. For MLEC-QA, we select samples from Clinic, Public Health, and Stomatology and exclude the rest. Examples can be found in Table 2.
Topic filtering To ensure that the collected data is related to the domain of maternity and infant care, we filter the samples according to their topics. We rely on four existing word-lists to determine the topic of each sample; the details can be found in Appendix B.3. We first translate the English word-lists into Chinese and then aggregate all the word-lists into a single one. To ensure relevancy, we manually check each word and only keep those ones that are exclusively related to the topic. In total, the aggregated word-list contains 238 Chinese domain-related words after dedduplication. We then use the aggregated word-list to perform the domain filtering. For KG datasets, we only keep triplets where both head and tail are included in the aggregated word-list. For MCQA datasets, we retain only the samples that contain at least one domain-related word from the aggregated word-list, either in the question or in the candidate answers. We further carry out dedduplication for the samples from MEDQA and MLEC-QA as they originate from the same source; here we only remove the duplicated samples from MLEC-QA and leave MEDQA untouched. In total, we obtain 79 samples from BIOS, 497 samples from CPubMed, 1,634 samples from MEDQA and 1,349 from MLEC-QA; these samples serve as the initial set for the subsequent benchmark construction.
$^{6}$ https://en.cma.org.cn/
3 CARE-MI benchmark
We construct the CARE-MI benchmark on top of the samples acquired in Section 2. The benchmark is based on two parts: a synthetic data generation process and a set of judgment models. With the synthetic data generation process, we create samples in the desired format for misinformation evaluation on LLMs. Then, empowered by the judgment model (see Section 4.3), we offer an automated metric for efficient and accurate LF generation evaluation, aiming to simplify human-based evaluation which is not only expensive but time-consuming. Note that both the synthetic data generation process and the judgment model construction are domain-agnostic and can be easily applied to other misinformation evaluation topics. In total, CARE-MI benchmark contains 1,612 samples; it is intended for LF generation evaluation under zero-shot [Wei et al., 2022, Brown et al., 2020b] setting. Statistics regarding the question length can be found in Figure 1. More details can be found in Appendix B.1.
Figure 1: Statistics of the questions in CARE-MI. Left: average question lengths for each source (average over all questions is shown in gray). Right: question length distribution.
The synthetic data generation process is summarized in Figure 2. It consists of five components: 1) true statement generation, 2) false statement generation, 3) question generation, 4) knowledge retrieval, and 5) expert annotation.
True statement generation We define true statements as sentences that are evidence-based and factually correct, for example, Placenta previa may potentially lead to preterm birth. We generate true statements from the samples collected in Section 2. For samples from KG datasets, we build the true statements heuristically from the triplets with rule-based methods. For samples from MCQA datasets, we formalize the generation of true statements as a QA2D task where the model is required to perform sentence transformations to combine the question and answer into a declarative sentence [Demszky et al., 2018]. The generation is done by using the combination of the rule-based method and an off-the-shelf model for simple cases such that the question from the QA pair ends with a pre-defined set of tokens such as "是" (referred as  $is$  in English), we directly concatenate the question with the answer as the synthetic true statement; otherwise, the generation is done using the GPT-3.5-turbo [OpenAI, 2023a]. Details regarding the implementation of the rule-based system can be found in our code repository; details about the prompts for true statement generation are in Appendix B.8.
False statement generation Similar to the definition of true statement, we define false statements as sentences that are factually incorrect, for example, progesterone is not a hormone. We approach the construction in two different ways: negation and replacement, corresponding to two types of false statements. For negated false statements, the generation is done on all available generated true statements where we generate the false statements by directly performing negation on the corresponding true ones; this construction procedure is also done by combining a rule-based method with applying the GPT-3.5-turbo and the details can be found in our code repository. We only generate false statements with replacement for samples that originally come from MCQA datasets; we generate the false statements by replacing the correct answers in the generated true statements with randomly selected wrong answers from the corresponding MC options. Our prompts utilized for generating negated statements can be found in Appendix B.7.
Question generation We generate questions based on the true statements. We rely on LLMs as their instruction-following ability allows us to generate questions efficiently instead of using resource-consuming fine-tuning methods. To select the best LLM for question generation, we experimentally
Figure 2: CARE-MI construction process. Data generation components are shown on green background, whereas the final benchmark samples are shown on orange background.
compare three LLMs available in Chinese: GPT-3.5-turbo, ChatGLM-6B [Du et al., 2022a, Zeng et al., 2022] and ChatYuan [ClueAI, 2023]. For this experiment, we evaluate the model performance on the BIOS dataset (See Appendix B.4 for more details) and select ChatYuan as the final choice.
Figure 3: An example of generated questions in MLEC-QA datasets.  $ZH$  and  $EN$  stands for Chinese and English. English sentences are translated for reference.
We generate two types of questions: True/False (TF) questions and open-ended (OE) questions. TF questions only admit binary answers, either agreeing or disagreeing with the statement in the question, while OE questions allow a variety of response styles. More specifically, we use ChatYuan along with a rule-based method to generate both TF and OE questions for MCQA samples while we only generate TF questions for KG samples (See Appendix B.4 for details). Information about the generated questions is shown in Table 3. In total, we generate 2,240 and 2,963 questions for MLEC-QA and MEDQA datasets, and 79 and 497 questions for BIOS and CPubMed datasets, respectively. Figure 3 shows an example of generated questions in the MLEC-QA dataset.
Table 3: Number of generated questions for CARE-MI.
Source	TF	OE
BIOS	79	-
CPubMed	497	-
MLEC-QA	1,333	907
MEDQA	1,617	1,346
Total	3,526	2,253
Knowledge retrieval Maternity and infant care is a critical subject matter wherein the dissemination of misinformation could endanger lives. In our benchmark, we include external knowledge in each sample to provide auxiliary information, not only for models but also for humans, allowing suitable
inferences on the veracity of the statements to be made. To achieve this, we apply BM25 [Robertson and Zaragoza, 2009] to obtain relevant knowledge from given knowledge sources based on the queries. We use two sources for knowledge bases: the Chinese Wikipedia<sup>7</sup> and Medical books collected by Jin et al. [2020]. We conduct topic filtering for Chinese Wikipedia as it originally contains a huge amount of pages that might not be relevant to the topic of maternity and infant care. During the knowledge retrieval, queries are the concatenation of the questions and the corresponding true statements. We conduct the retrieval on paragraph level; to account for differences between the two knowledge sources, we retrieve the top three most relevant paragraphs from both sources, respectively.
Expert annotation We hire two medical-domain experts as annotators to review the generated samples and a third meta-annotator to arbitrate their disagreement. The annotators are instructed to answer a series of guiding questions. The guideline including rules and procedures that annotators should follow as well as the inter-annotator agreements are detailed in Appendix A.1. We ask them to evaluate 5,779 synthetic samples, and we discard those that receive two or more negative evaluations. After the expert annotation, the benchmark is downsized to 1,612 samples.
We evaluate Chinese LLMs with respect to misinformation in LF generation using our proposed benchmark. Experiments are done in the form of single-round conversation where we feed models with questions and collect the direct output from the corresponding model. We perform the evaluation under the true zero-shot [Lin et al., 2022] setting: for each model, the input is a question from our benchmark, without any additional instruction and examples; prompt and hyperparameters are not tuned in any way. All questions in CARE-MI are used in the evaluation.