SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1. Fluency: the sentence must be fluent in the Chinese language, e.g., it owns valid word order and reasonable sentence structure.
2. Consistency: the generated questions should ask about the same entities or events mentioned in the true statements.
3. Leakage prevention: a valid question should not directly contain the correct answer.
The generated question is considered incorrect if it fails in any of the above metrics.
We choose three LLMs available in Chinese for a preliminary comparative experiment: GPT-3.5-turbo [OpenAI, 2023a], ChatGLM-6B [Du et al., 2022b] and ChatYuan [ClueAI, 2023]. Results are shown in Table 12. Among them, ChatYuan has the best accuracy as well as the best efficiency, and thus is selected for the question generation task.
Generation of TF and OE questions Theoretically speaking, we can generate TF and OE questions for all samples. In our case, we consider the trade-off between employing a rule-based method and using a LLM; rule-based method allows for fast and accurate generation of statements with certain explicit patterns while a LLM can handle more complex patterns albeit with a slightly higher error rate. Since a large
Table 12: The result of the comparative pioneer experiment on BIOS dataset. Acc. stands for accuracy. The best performances are highlighted in bold.
Model	Acc.	Time (s)
GPT-3.5-turbo [OpenAI, 2023a]	0.77	74
ChatGLM-6B [Du et al., 2022b]	0.28	433
ChatYuan [ClueAI, 2023]	0.79	56
portion of the true statements from KG samples do not have explicit patterns, we omit the generation of OE questions for them. However, we believe that this compromise could be resolved with a more carefully designed prompt instruction, even though it might lead to higher cost.
B.5 Experimental settings
Chinese LLM evaluation For evaluation, we limit all models to generate a maximum of 2,048 tokens. To restrain the variation during generation, we set the temperature as low as possible; for the GPT family models, we set the temperature to 0; otherwise it is set to 0.01. For decoding, we apply greedy strategy for all evaluated models. We apply the default parameters for the rest of the settings.
Judgment model We fine-tune the judgment models to mimic expert evaluation on both aspects, correctness and interpretability. We experiment with BERT-Large, GPT-3-350M, GPT-3-6.7B and LLaMA-13B-T. LLaMA-13B-T is obtained by further pretraining the LLaMA-13B on additional corpus for 250 hours and fine-tuning on selected instruction-following tasks for 170 hours on 8 NVIDIA A100 GPUs. We use the OpenAI API to fine-tune GPT-3-350M and GPT-3-6.7B. For correctness, the cost is  $30.46 and$ 228.48, and for interpretability, the cost is $30.56 and $229.20 correspondingly. Time utilized for fine-tuning LLaMA-13B-T is approximately 3 hours on both correctness and interpretability. We set the temperature to 0 for the GPT models and 0.01 for LLaMA-13B-T to ensure that the output is consistent. The maximum token length is 2,048 for the GPT models and 512 for BERT-Large; to ensure that the maximum length requirement is satisfied, the knowledge sentences are truncated correspondingly.
B.6 Experiment result statistics
Table 13: Average number of sentences and tokens.
Models	All	BIOS	CPubMed	MLEC-QA	MEDQA
Average number of sentences
MOSS-16B-SFT [Sun and Qiu, 2023]	4.591	4.727	4.589	4.390	4.662
ChatGLM-6B [Du et al., 2022b]	6.814	6.318	7.153	6.917	6.866
BELLE-7B-2M [Ji et al., 2023b]	3.546	3.636	3.973	3.355	3.218
BELLE-7B-0.2M [Ji et al., 2023b]	2.240	2.046	2.100	2.367	2.447
GPT-4 [OpenAI, 2023b]	4.922	4.818	5.367	4.741	4.762
GPT-3.5-turbo [OpenAI, 2023a]	4.066	4.046	4.447	3.896	3.874
LLaMA-13B-T [Touvron et al., 2023]	5.863	5.409	5.827	6.109	6.107
All	4.577	4.429	4.779	4.539	4.562
Average number of tokens
MOSS-16B-SFT [Sun and Qiu, 2023]	115.418	100.546	116.640	119.120	125.365
ChatGLM-6B [Du et al., 2022b]	224.395	185.091	238.013	236.917	237.558
BELLE-7B-2M [Ji et al., 2023b]	83.895	85.318	99.293	78.892	72.076
BELLE-7B-0.2M [Ji et al., 2023b]	33.841	20.727	29.333	41.199	44.102
GPT-4 [OpenAI, 2023b]	134.475	118.636	148.953	134.799	135.510
GPT-3.5-turbo [OpenAI, 2023a]	98.027	90.591	110.247	96.071	95.202
LLaMA-13B-T [Touvron et al., 2023]	171.671	133.000	163.753	191.930	198.002
All	123.103	104.844	129.462	128.418	129.688
We report the average number of sentences and average number of tokens for each answer from each evaluated models regarding the question sources. The statistics are presented in Table 13. Unlike
in MCQA evaluation where the models only generate few tokens, the answers generated during our evaluation contain an average of 4.6 sentences and 123.1 tokens.
B.7 Prompts for negated statement generation
We also apply GPT-3.5-turbo [OpenAI, 2023a] to generate negated statements, with the temperature set to 0. The prompts are shown below.
Negated statement generation: System prompt
Given the following sample, generate the negated declarative sentences.
Negated statement generation: User prompt
You are an accurate NLP annotator.Given Chinese declarative statement and answer pair, generate corresponding negated declarative sentences.Do the least modification during the generation.Make sure that the generated negated declarative sentences are fluent.For example:S:比较甲、乙两地新生儿的死因构成比，宜绘制圆图.N:比较甲、乙两地新生儿的死因构成比，不宜绘制圆图.S:行人工破膜后9小时宫口开9cm提示活跃期延长.N:行人工破膜后9小时宫口开9cm不提示活跃期延长.S:胎儿和婴幼儿期生长遵循头尾发展律.N:胎儿和婴幼儿期生长不遵循头尾发展律.S:治疗该病，目前首选阿昔洛维.N:治疗该病，目前不首选阿昔洛维.S:习惯性晚期流产最常见于子宫颈内口松弛.N:习惯性晚期流产不常见于子宫颈内口松弛.S:胎头最低点在坐骨棘水平说明胎头已经衔接.N:胎头最低点在坐骨棘水平不说明胎头已经衔接.Now, given the following sample, generate the negated declarative sentences:
B.8 Prompts for true statement generation
To leverage LLM for true statement generation, we need to carefully calibrate the prompt instruction. We form this generation as a QA2D task where we generate the statements using the QA pairs. In many cases, the question is too long; on the one hand, such long sequence input might lead to unstable generation from LLM; on the other hand, it is not necessary to utilize the whole question for statement generation as usually only the last sub-sentence is strongly related to the desired statement when doing the QA2D task. Thus, we only use the last sub-sentence and the corresponding answer to generate the statement. We use GPT-3.5-turbo [OpenAI, 2023a] for generating true statement, with the temperature set to 0. the prompts are shown in below:
True statement generation: System prompt
Given Chinese question and answer pair, combine and modify them to produce corresponding declarative sentences.
True statement generation: User prompt
You are an accurate NLP annotator.Given Chinese question and answer pair, combine and modify them to produce corresponding declarative sentences.Kep the information mentioned in the Chinese question and answer pair unchanged.Do the least modification during the generation.Make sure that the generated declarative sentences are fluent.For example:Q:淋病是何种类型的炎症A:急性化脓性炎症D:淋病是急性化脓性炎症Q:下肢浮肿(+)A:轻度妊高征D:下肢浮肿(+)提示轻度妊高征Q:严重肝功能不全的病人不宜用A:泼尼松D:严重肝功能不全的病人不宜用泼尼松Q:肠扭转引起的坏死A:湿性坏疽D:肠扭转引起的坏死是湿性坏疽Q:肾脏中抗原抗体复合物的检测A:免疫比浊法D:肾脏中抗原抗体复合物的检测应使用免疫比浊法Q:两侧坐骨棘间径≥10cmA:女型骨盆D:两侧坐骨棘间径≥10cm说明是女型骨盆Q:此时局部最佳处理方法A:冲洗上药D:此时局部最佳处理方法是冲洗上药Q:应当对A:孕妇进行产前诊断D:应当对孕妇进行产前诊断Q:是因为母乳中A:含白蛋白、球蛋白较多D:是因为母乳中含白蛋白、球蛋白较多Q:治疗24小时后仍有自觉症状A:剖宫产D:治疗24小时后仍有自觉症状,应采取剖宫产Q:月经期使用清洁卫生巾A:避免感染D:月经期使用清洁卫生巾可避免感染Q:1小时后儿头下降0.5cmA:胎头下降延缓D:1小时后儿头下降0.5cm提示胎头下降延缓Q:子宫出现Hegar征A:孕6周时开始D:子宫出现Hegar征从孕6周时开始Now, given the following sample, generate the declarative sentences:
B.9 Prompt for fine-tuning the judgment models
For aspects including correctness and interpretability, we use the following prompts to fine-tune judgment models:
Judgment fine-tuning: [aspect]
听取某AI助手对一个医学问题的回答，并在[aspect]方面对其进行打分，不用解释原因。问题：[question]参考资料：[knowledge1][knowledge2]...[knowledgen]某AI助手：[answer]你的评分：
Here  $n \geq 3$  as each sample in our benchmark is guaranteed to have at least 3 pieces of supporting knowledge. The prompts are utilized for the fine-tuning of GPT-3-350M, GPT-3-6.7B and LLaMA-13B-T. All prompts utilized here are not tuned.