SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Models As our goal is to evaluate the LLM misinformation in LF generation, we focus on the evaluation of autoregressive language models that are trained for generation tasks, as opposed to masked language models like BERT [Devlin et al., 2019] or RoBERTa [Liu et al., 2019]. We evaluate Chinese LLMs that have been specifically tuned for conversation scenarios, either by supervised fine-tuning (SFT) or reinforcement learning with human feedback (RLHF) as we aim at measuring the misinformation occurring when the models directly interact with humans. We include ChatGLM-6B [Zeng et al., 2022, Du et al., 2022a], MOSS-16B-SFT [Sun and Qiu, 2023], two variants of the BELLE [Ji et al., 2023b,c] series, a.k.a BELLE-7B-0.2M and BELLE-7B-2M, and two models from the GPT family, GPT-3.5-turbo [OpenAI, 2023a] and GPT-4 [OpenAI, 2023b]. We also include a LLaMA [Touvron et al., 2023] model which we perform further pretraining on Chinese corpus including CLUECorpus2020 [Xu et al., 2020b], Chinese Scientific Literature Dataset [Li et al., 2022], Chinese Wikipedia<sup>7</sup>, RedPajama-Data [Computer, 2023], etc., and fine-tuning on translated ShareGPT [RyokoAI, 2023], Aplaca-GPT4 [Peng et al., 2023] and WizardLM [Xu et al., 2023]. We denote this version as LLaMA-13B-T. More details regarding the parameter settings can be found in Appendix B.5. Finally, we recruit a domain expert to act as a human baseline; we randomly select 200 questions from the benchmark and let the expert answer them correspondingly; the selected samples are collected strictly following their original distribution in the benchmark regarding their sources. The expert is allowed to check any resource that is necessary and is suggested to finish each question within 2 minutes.
Evaluation metrics Unlike Lin et al. [2022] who consider an answer to be truthful if and only if it avoids asserting a false statement and thus allows non-committal answers such as No comments and I don't know as legal truthful answers, we follow the human evaluation framework similar to what has been explored in [Lu et al., 2022]. For each model-generated answer, we recruit three medical-domain expert-level annotators to evaluate the following two aspects:
1. Correctness: given a question, whether the answer is factually correct and relevant.
2. Interpretability: given a question, whether the answer contains a detailed and concise explanation that demonstrates how the conclusion was reached.
<sup>7</sup>https://dumps.wikimedia.org/zhwiki/
Table 4: Average correctness scores for all models and human baseline. Deeper color indicates better performance. We only randomly select 200 questions for human evaluation.
Model	All	BIOS	CPubMed	MLEC-QA	MEDQA
MOSS-16B-SFT (2023)	0.671±0.321	0.930±0.121	0.925±0.166	0.644±0.332	0.639±0.316
ChatGLM-6B (2022b)	0.610±0.333	0.928±0.116	0.748±0.264	0.579±0.346	0.599±0.328
BELLE-7B-2M (2023b)	0.647±0.315	0.843±0.268	0.928±0.175	0.631±0.314	0.605±0.311
BELLE-7B-0.2M (2023b)	0.670±0.316	0.947±0.095	0.942±0.141	0.624±0.335	0.646±0.302
GPT-4 (2023b)	0.867±0.215	0.958±0.125	0.967±0.124	0.851±0.233	0.858±0.211
GPT-3.5-turbo (2023a)	0.824±0.263	0.973±0.108	0.948±0.160	0.799±0.279	0.815±0.263
LLaMA-13B-T (2023)	0.709±0.301	0.871±0.235	0.922±0.178	0.678±0.311	0.689±0.297
Human Baselined†	0.938±0.213	1.000±0.000	1.000±0.000	0.945±0.196	0.908±0.262
Table 5: Average interpretability scores for all models. Deeper color indicates better performance.
Model	All	BIOS	CPubMed	MLEC-QA	MEDQA
MOSS-16B-SFT (2023)	0.746±0.229	0.920±0.115	0.883±0.154	0.726±0.245	0.731±0.222
ChatGLM-6B (2022b)	0.730±0.251	0.929±0.112	0.779±0.248	0.705±0.263	0.734±0.242
BELLE-7B-2M (2023b)	0.728±0.235	0.839±0.251	0.930±0.140	0.723±0.236	0.694±0.228
BELLE-7B-0.2M (2023b)	0.645±0.237	0.716±0.138	0.746±0.111	0.609±0.266	0.650±0.229
GPT-4 (2023b)	0.928±0.134	0.973±0.083	0.981±0.060	0.921±0.146	0.922±0.133
GPT-3.5-turbo (2023a)	0.883±0.178	0.977±0.073	0.960±0.094	0.864±0.201	0.880±0.171
LLaMA-13B-T (2023)	0.816±0.200	0.836±0.265	0.935±0.127	0.797±0.214	0.808±0.192
We require the annotators to make their judgments independently during the evaluation; for each aspect, they are asked to give a scalar score between 0 and 1 to reflect their decisions for each sample (the higher the better). The final evaluation results are averaged over the three annotators. We refer the readers to Appendix A.2 for more details about this human evaluation.
Overview Evaluation results on correctness and interpretability are shown in Table 4 and Table 5, respectively. Among all evaluated Chinese models, models from the GPT family perform the best by a large margin in both correctness  $(\geq 0.158)$  and interpretability  $(\geq 0.112)$ . In general, all models exhibit better performance on samples generated from KG datasets than from MCQA datasets, as they generally have longer context and necessitate more reasoning ability from the models to produce the correct answer. Overall, LLMs with smaller sizes tend to perform worse; however, even the best model is not comparable with human expert, indicating room for improvement. Figure 4 further presents the correctness and interpretability evaluation for TF and OE questions separately. We observe that all models perform better on TF questions than OE questions: on average, most models are able to achieve 0.8 of correctness as well as interpretability for TF questions while only GPT models can achieve a correctness of 0.6 for OE questions. This highlights the weakness of current models, where LLMs still struggle with complex reasoning.
Correctness is linear-correlated with interpretability Figure 5 shows the correlation between correctness and interpretability across all evaluated LLMs. In general, all models exhibit a similar pattern, where the interpretability linear correlates with the corresponding correctness. Exceptions include BELLE-6B-0.2M, which shows abnormally low interpretability. On the contrary, ChatGLM-6B and LLaMA-13B-T present above-average interpretability; however, this is not always good. Better interpretability scores only indicate better generated descriptions when explaining how the conclusions are drawn (See Section 4.1); yet if the conclusions themselves are factually incorrect, more detailed explanations will only lead to misleading consequences.
More data is not always better BELLE-7B-0.2M present slightly better performance  $(\uparrow 0.023)$  but lower interpretability  $(\downarrow 0.083)$  in comparison with its twin model BELLE-7B-2M. Both BELLE-7B-0.2M and BELLE-7B-2M use BLOOM [Scao et al., 2022] as base model with the only difference being the size of the instruction set used during fine-tuning. As mentioned, better interpretability is not always a desired property, especially when it is paired with much lower correctness; more instruction
Figure 4: Evaluation results for TF and OE questions.
tuning yields better language ability, yet it has no effect on improving knowledge correctness. This indicates that increasing the size of instructions used for fine-tuning LLMs is not always helpful in improving its knowledge capability. The conclusion is intuitive: the objective for fine-tuning does not align with improving truthfulness. Consequently, increasing the size of instruction-following samples will not assist the model in providing more truthful answers.
4.3 Automated metrics
Human evaluation, especially in knowledge-intensive domains, is costly and difficult to reproduce. On the other hand, traditional automated evaluation metrics, such as perplexity [Dinan et al., 2019], BLEU score [Khashabi et al., 2021a] and ROUGE score [Lin, 2004], suffer from a misalignment problem in which the metrics fail to capture the actual performance of the models. Lin et al. [2022] propose to train a judgment model using human labels to serve as a proxy of human annotators. However, the trained models cannot transfer across different tasks and languages. To enable efficient and ac
curate automated evaluation for our proposed benchmark, we explore using different architectures as backbones of the judgment models and fine-tune them to mimic human judgment.
Figure 5: Correctness and interpretability metrics show a linear relationship with the  $R^2$  being 0.834.
Table 6: Pearson correlation scores (Pear.) and accuracy (Acc.) are reported for each trained judgment model. We highlight the best Pearson correlation scores in bold.
Metric	Random	BERT-Large	GPT-3-350M	GPT-3-6.7B	LLaMA-13B-T
Pear.	Acc.	Pear.	Acc.	Pear.	Acc.	Pear.	Acc.	Pear.	Acc.
Correctness	-	0.560	0.606	0.560	0.783	0.835	0.803	0.858	0.868	0.898
Interpretability	-	0.800	0.013	0.794	0.565	0.822	0.634	0.828	0.683	0.835
We use 1) both generated true and false statements as synthetic positive and negative answers, and 2) the human evaluation results, to train all judgment models. We also feed models with all available retrieved knowledge. To evaluate the performance of each judgment model, we select the answer outputs from MOSS-16B-SFT and ChatGLM-6B as the validation set. This is because the scores these two models received are closer to 0.5 on average, indicating a more balanced distribution of good and bad answers. We perform the training using the answers from the rest of the evaluated models. In order to compare different architectures, we fine-tune BERT-Large [Devlin et al., 2019], two GPT-3 variants (GPT-3-350M and GPT-3-6.7B) and LLaMA-13B-T (See Section 4.1 for details). For training BERT-Large, we use the [SEP] token as the separator between fields and directly
concatenate all fields together (e.g., question, answer and retrieved knowledge) as the input. Prompt used for fine-tuning LLMs can be found in Appendix B.9. For BERT-Large and the two GPT variants evaluated, as they suffer from input length limitation, we truncate the retrieved knowledge to make sure the input meets the model requirements. For comparison, we also include a majority baseline, which always predicts the most frequent score in the training set. More details regarding the experiment settings can be found in Appendix B.5. We evaluate the performance of the judgment model using Pearson correlation and accuracy. To obtain accuracy, we cast the scalar output of the judgment models into binary labels by empirically setting the threshold to 0.5. Judgment models' results are shown in Table 6.
We observe that BERT-Large does not learn anything useful. This might be due to the stringent constraints on the input size as well as the model's inherent inability in complicated reasoning tasks. Larger models tend to perform better; however, it is harder for models to understand the correlation between input text and the labels in the aspect of interpretability. For the final judgment model, we select LLaMA-13B-T as the backbone, as it obtains the best results. We train the model again but with all available data, including both training and validation samples.
Table 7: Evaluations are done on the validation set.  $w / o$ $K$  . and  $w/ K$  . represents fine-tuning the models without and with knowledge. We highlight the best Pearson correlation scores in bold.
Metric	w/o K.	w/ K.
Pear.	Acc.	Pear.	Acc.
Correctness	0.779	0.806	0.868	0.898
Interpretability	0.639	0.867	0.683	0.835
We make the judgment models for both aspects publicly available.
We further conduct an ablation study to assess whether incorporating retrieved knowledge enhances the performance of judgment models. Specifically, we perform a comparison by training the LLaMA-13B-T both with and without the retrieved knowledge during the fine-tuning while maintaining the rest of the settings identical. The experiment results are shown in Table 7, which demonstrate that adding knowledge improves the performance of the judgment models.
In this paper, we proposed CARE-MI, a Chinese benchmark for LLM misinformation evaluation in LF generation in the topic of maternity and infant care. This is the first Chinese benchmark that aims to quantify the misinformation in LF generation cases, the only dataset with clear expert-annotations on the maternity and infant care domain, and with a construction pipeline that can be easily transferred to other domains and languages. We conducted comprehensive assessments on Chinese LLMs, and showed that current models still have room for improvement. Furthermore, we investigated different model backbones for training judgment models and provided an efficient judgment model that can perform correctness evaluation of LF generation accurately. We believe that our proposed benchmark not only paves the way for easier benchmark construction for the whole research community, but also contributes to promoting better Chinese LLM applications in the maternity and infant care domain.
**Limitation** CARE-MI aims solely at evaluating the misinformation in long-form generation tasks for Chinese LLMs on the topic of maternity and infant care. It is not designed for any other scenarios, e.g., evaluating other target groups such as medical professionals or students. The misuse of CARE-MI might lead to unpredictable consequences. Additionally, the benchmark contains content that can only be considered as correct for now. With the development of modern clinical techniques, we expect the accuracy of the information provided in the benchmark to decrease over time, and thus our proposed benchmark might not be suitable for misinformation evaluation at a later stage, e.g., 10 years from now. Furthermore, our benchmark might not align with the actual interests of the maternity and infant care community (e.g., pregnant women) in real-life situations, as the benchmark is not constructed by collecting the most frequently asked questions in the community. Last but not least, even though we have tried our best to reduce the subjective bias during the human annotations, we cannot completely avoid it.
This work was partly supported by JSPS KAKENHI No. JP22K12091.
Together Computer. Redpajama: An open source recipe to reproduce llama training dataset, 2023. URL https://github.com/togethecomputer/RedPajama-Data.
RyokoAI. ShareGPT52K, 2023. URL https://huggingface.co/datasets/RyokoAI/ShareGPT52K.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction Tuning with GPT-4. CoRR, abs/2304.03277, 2023. doi: 10.48550/arXiv.2304.03277. URL https://doi.org/10.48550/arXiv.2304.03277.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardLM: Empowering Large Language Models to Follow Complex Instructions. CoRR, abs/2304.12244, 2023. doi: 10.48550/arXiv.2304.12244. URL https://doi.org/10.48550/arXiv.2304.12244.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 320-335. Association for Computational Linguistics, 2022b. doi: 10.18653/v1/2022.acl-long.26. URL https://doi.org/10.18653/v1/2022.acl-long.26.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Galle, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoit Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunj Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, and et al. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model. CoRR, abs/2211.05100, 2022. doi: 10.48550/arXiv.2211.05100. URL https://doi.org/10.48550/arXiv.2211.05100.
Joseph L Fleiss. Measuring nominal scale agreement among many raters. Psychological bulletin, 76 (5):378, 1971.
A Annotation guidelines
A.1 Benchmark expert annotation
We hire experts as our annotators in order to filter and improve the quality of our generated data. Each sample contains a generated sample question, a generated true statement serving as the default correct answer to the question, a generated false statement serving as the default wrong answer, and a few paragraphs serving as the supporting knowledge. To make judgment for a sample, annotators are presented with a set of guiding questions; each annotator is required to make judgment on these guiding questions and make decision to each of them independently.
Table 8: Fleiss' kappa between the two annotators. Here  $Q$  . index represents the index of the guiding questions.
Q. index	Fleiss' kappa	Interpretation
1	0.657	Substantial agreement
2	0.578	Moderate agreement
3	0.818	Almost perfect
4	0.786	Substantial agreement
Annotators are asked to make judgments on the guiding questions in the order they are presented. Specifically, for each sample, the annotators should consider the following guiding questions:
1. Judge whether the sample question has at most one single correct answer. $^{8}$
2. Judge whether the description or claim made in the sample question aligns with medical norms and is non-controversial.
3. Judge whether the synthetic true answer to the sample question is factually accurate and directly answers the question.
4. Judge whether the knowledge contains any relevant information from which the correct answer to the sample question can be directly derived or indirectly implied.
For each guiding question mentioned above, the annotators are required to assign 1 to it if the corresponding requirements are properly met, otherwise 0.
We hire three experts for this annotation; one of them serves as the meta-annotator for the purpose of resolving disagreements between the rest of the annotators. In principle, the meta-annotator will only need to make judgments on specific guiding questions where disagreements arise. All the annotators are required to make judgment on each sample independently, but with access to any necessary resources. All annotators are suggested to finish the annotation for each sample within 5 minutes. We report Fleiss' kappa [Fleiss, 1971] between the two annotators and present the agreement scores in Table 8.
We collect the annotations from the expert and set thresholds in order to select a relatively high quality portion of the generated samples while maintaining an appropriate benchmark
size. Originally we have 5,779 samples; from them, we select 1,624 samples that 1) have at most one single correct answer, 2) have a clear description or claim in the sample question which aligns with medical norm and is non-controversial, 3) the generated answer to the question is factually correct and directly answer the sample question, and 4) have at least 3 pieces supporting knowledge (e.g.,  $N_K \geq 3$ ), and exclude the rest leveraging the above mentioned annotation procedure. We further
Figure 6: Size of samples remained after applying different thresholds.  $N_K$  stands for the number of supporting knowledge.
<sup>8</sup>We believe that, even though an answer to the sample question can have various superficial forms, the facts/information behind all true answers should be the same. So we are not restricting the answers by their forms, but the content that they contain.
exclude 12 samples that are not linguistically fluent from the remaining samples and eventually obtain 1,612 samples for the benchmark. Details about the expert data filtering results are presented in Figure 6.
A.2 Misinformation human evaluation
We ask the experts to do misinformation evaluation of Chinese LLMs in the topic of maternity and infant care domain using the sample questions from our proposed benchmark. We hire three experts to evaluate the answers from the Chinese LLMs independently; unlike the previous annotation, we don't have a meta-annotator in this part. For each LLM answer, the annotators are asked to evaluate on two different aspects: correctness and interpretability. Annotators should assign a scalar between 0 and 1 to a sample for each criterion. Heuristic evaluation guidelines for each criterion are presented in Table 9. To evaluate the inter-annotator agreement, for both aspects, we first cast the scalar scores into four categories, i.e., Very Low for scores between 0 and 0.25, Low for scores between 0.25 and 0.5, High for scores between 0.5 and 0.75, and Very High for scores between 0.75 and 1. Under this setting, the Fleiss' kappa for correctness is 0.755 (substantial agreement) and 0.573 (moderate agreement) for interpretability.
Table 9: Guidelines for evaluating the answers from LLMs given the benchmark questions. This guideline serves as the heuristic rules which all annotators should follow.
Metric	Range	Description
Correctness	0.75 - 1.0	The answer is in general factually correct, clear and directly resolving the question.
0.5 - 0.75	The answer contains some non-rigorous content, but directly addresses the question.
0.25 - 0.5	The answer contains multiple descriptions or claims that are either incorrect or incomplete, but the correct answer can be inferred from the current answer.
0.0 - 0.25	The answer contains a large amount of incorrect and incorrect descriptions or claims, and the correct answer is unable to be inferred from the current answer.
Interpretability	0.75 - 1.0	All descriptions or claims in the answer are relevant and reasonable to the conclusion drawn by the answer.
0.25 - 0.75	The answer contains some descriptions or claims that are either unreasonable or irrelevant to the conclusion drawn by the answer.
0.0 - 0.25	The answer contains many descriptions or claims that are unreasonable and irrelevant to the conclusion drawn by the answer.
B Details of CARE-MI
B.1 Benchmark Statistics
We additionally present benchmark statistics regarding the evidence length in Figure 7. We observe that the number of retrieved knowledge from MLEC-QA and MEDQA has a incredibly similar distribution. questions collected from BIOS are accompanied with the largest number of knowledge, as most of them (around  $40\%$  ) have 6 pieces of valid supportive knowledge.
Figure 7: Left: the average number of knowledge for each source (average over all knowledge is shown in gray). Right: number of knowledge distribution for each source.
B.2 KG triples
We present examples of the KG triples in Table 10.
Table 10: Examples of Chinese triples in BIOS and CPubMed. English translations are shown for reference.
Source	Head	Relation	Tail
BIOS	子宫内膜异位症
endometriosis	可导致
can lead to	呕吐症状
vomiting
治疗性催产素
therapeutic oxytocin	是一种
is a	激素
hormone
CPubMed	唐氏综合症
down syndrome	影像学检查
radiological examination	B超
B ultrasound
前置胎盘
placenta previa	可导致
can lead to	产前出血
antepartum hemorrhage
B.3 Topic filtering
We use an aggregated word-list to filter maternity and infant care domain related samples. Details about the word-list can be found in Table 11.
Table 11: The word-lists that we have utilized for finding maternity and infant care related samples. The aggregated word-list presented here is filtered and deduplicated.
Source	Language	Size
The Women’s Health Group	English	87
Department of Health, State Government of Victoria, Australia	English	99
Maternal and Infant Care Clinic	English	57
Having a Baby in China	Chinese/English	267
Aggregated word-list	Chinese	238
B.4 Question generation
Choice of QG model We construct questions on top of generated true statements using LLMs for their ease of usage; the true statements should naturally serve as the correct answers to the generated questions, which requires that the generated questions refer to the same entities or events mentioned in the statements. We select the BIOS dataset for question generation model comparison and evaluate candidate models through a set of metrics: