RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
B.1 Proof of Theorem 3.1
We omit the case when FI_{C}=0 since in that case, we cannot estimate TrueE . By the definition in Theorem 3.1 , we have S_{C} generated answers that satisfy C . Let’s denote k=S_{C} for simplicity. Let’s denote k performance scores of answers satisfying C as x_{1},\cdots,x_{k} , and \bar{x}=\frac{\sum_{i=1}^{k}(x_{i})}{k} as the mean. Finally, TrueE is the population mean of the performance scores, denoted as \mu .

Statement 1: EstTrueE is consistent. From Eq. 5 , by rewriting EstTrueE , we have EstTrueE=\frac{1}{n}\cdot\sum_{i=1}^{k}(x_{i})\cdot\frac{n}{k}=\bar{x} , which is an unbiased estimator of the average performance TrueE , i.e., Bias(\bar{x})=0 or \lim_{k\to\infty}Bias(EstTrueE)=0 (1). Now, let’s denote the variance of the performance scores as \sigma^{2} , then the variance of EstTrueE is Var(EstTrueE)=Var(\bar{x})=\frac{\sigma^{2}}{n} and \lim_{k\to\infty}Var(EstTrueE)=0 (2). From (1) and (2), by the Sufficient Condition for Consistency Amemiya ( 1985 ) , we conclude that EstTrueE is a consistent estimator.

Statement 2: FI_{C} value. Let’s denote s^{2}=\frac{1}{k-1}\sum_{i=1}^{k}(x_{i}-\bar{x})^{2} as the sample variance of the performance scores x_{i} s. It is well-known that \frac{\sqrt{k}(\bar{x}-\mu)}{s}\sim t_{k-1} . For estimating the population mean \mu with finite population size n and the type I error \alpha , we have the margin of error \epsilon :

\epsilon\geq t_{\alpha/2,k-1}\cdot\sqrt{\frac{n-k}{n}\cdot\frac{s^{2}}{k}} (9)

where \frac{n-k}{n} is the finite population correction factor. Eq. 9 is equivalent to:

k\geq\frac{n-k}{n}\cdot\left(\frac{t_{\alpha/2,k-1}\cdot s}{\epsilon}\right)^{2} (10)

which yields

k\geq\frac{1}{\frac{1}{n}+\left(\frac{\epsilon}{t_{\alpha/2,k-1}\cdot s}\right)^{2}}

then

FI_{C}=\frac{k}{n}\geq\frac{1}{1+n\cdot\left(\frac{\epsilon}{t_{\alpha/2,k-1}\cdot s}\right)^{2}}

Statement 3: When FI_{C} approaches 1 , EstTrueE approaches TrueE . Since EstTrueE by its definition in Eq. 5 is continuous with respect to FI_{C} ( Eq. 5 ), S_{C} ( Eq. 3 ) and F_{C} ( Eq. 3 ), therefore, we have the equality:

\lim_{{FI_{C}}\to 100\%}{(EstTrueE)}=EstTrueE(FI_{C}=100\%)={TrueE}.

∎
C.1 Multiple-choice Question (MCQ) Discussions
We evaluate Gemma, Mistral, and ChatGPT on the MMLU and BBH datasets using two prompting techniques, Zero-shot (ZS) and Zero-shot Chain-of-Thought (ZS-CoT) ( Section 5.1 ). The prompts are specified in Section E.1 .
We report the FI_{C},SysE,EstTrueE scores. The results are presented in Tab. 2 . Additionally, Tab. 3 , Tab. 4 , and Tab. 5 are the distillation results of Tab. 2 :

Tab. 3 . For each model, we average its EstTrueE performance overall benchmarks and prompting techniques. For each task, we average the EstTrueE scores overall models and prompting techniques. The results of this table are plotted in Fig. 2 and discussed in Section 5.1 .

Tab. 4 . The purpose of this table is to compare the FI scores across formats. We average all the FI scores across models and tasks.

Tab. 5 . The purpose of this table is to see whether CoT Wei et al. ( 2022 ) mitigates format bias. We average all the EstTrueE scores over all models and benchmarks for each ZS and ZS-CoT prompting method.
Format instruction-following bias.
The FI scores across formats are illustrated in Tab. 4 . There is a notable difference between the scores for character-based and textual value-based formats. The score for character format (51.39%) is more than twice as high as that for textual value format (20.46%). Among the models, ChatGPT follows the instructions best with FI score 52.42\% . Below we present two examples of Gemma and Mistral failing to follow the format instructions:
ZS versus ZS-CoT.
We investigate the impact of Chain-of-Thought (CoT) Wei et al. ( 2022 ) on mitigating format bias. The average of estimated True Accuracy scores across models and tasks for the ZS prompting and ZS-CoT prompting are shown in Tab. 5 . ZS prompting achieves a higher score in the character category (51.20%) compared to ZS-CoT prompting (44.13%). Similarly, for textual value format, ZS prompting scores higher (20.83%) than ZS-CoT prompting (16.99%). However, the BiasF_{o} is lower for the ZS-CoT model ( 184.14\%^{2} ) compared to the ZS model ( 230.58\%^{2} ), indicating that CoT slightly decreases the format bias.
Reliability of the results.
From Tab. 2 , we see that 16/24 of the estimated EstTrue results are reliable. The reliability of results in the MCQ output format varies across different models. Gemma-7B-it and Mistral-7B-it show significant unreliability in textual value format, evidenced by numerous red-marked scores due to models not following the format instructions to output correct formats. In contrast, ChatGPT’s results are significantly more reliable in the MMLU and BBH benchmarks ( 7/8 ), with only one unreliable result in the BBH textual format output.
C.2 Wrapping Discussions
We examine Gemma, Mistral, and ChatGPT on the MCQ datasets (MMLU,BBH) and generation datasets (GSM8K, HotpotQA, FairytaleQA) utilizing two prompting techniques, Zero-shot (ZS) and Zero-shot Chain-of-Thought (ZS-CoT) ( Section 5.2 ). The prompts are also provided in Section E.2 . We measure the FI_{C},SysE,EstTrueE . The results are shown in Tab. 6 . Furthermore, Tab. 7 , Tab. 8 and Tab. 9 are the distillation outcome of Tab. 6 :

Tab. 7 . For each model, we average its EstTrueE performance overall benchmarks and prompting techniques. For each task, we average the EstTrueE scores overall models and prompting techniques. This table is plotted in Fig. 3 and discussed in Section 5.2 .

Tab. 8 . The purpose of this table is to compare the FI scores across formats. We average all the FI scores across models and tasks.

Tab. 9 . The purpose of this table is to see whether CoT Wei et al. ( 2022 ) mitigates format bias. We average all the EstTrueE scores over all models and benchmarks for each ZS and ZS-CoT prompting method.
Format instruction-following bias.
The FI scores over formats are provided in Tab. 8 . Overall, LLMs exhibit significant format-following bias across formats with a variance of FI scores of 297.28\%^{2} . Among the models, ChatGPT follows the instructions best with average FI Score 85.01\% . The “Special Character” wrapping format has the highest FI score of 73.34\% . Following it is the “Placeholder” wrapping format also shows a high FI score of 68.37\% , suggesting it is another effective format for ensuring instruction adherence. In contrast, the “Quoting” wrapping format has the lowest FI score of 17.06\% . This significant drop compared to other formats suggests that quoting is the least effective method for wrapping instructions, possibly causing confusion or misinterpretation by the models. Below we present two examples of Gemma and Mistral failing to follow the format instructions:
ZS versus ZS-CoT.
The average of estimated True Accuracy scores across models and tasks for the ZS and ZS-CoT prompting are shown in Tab. 9 . For the majority of the wrapping methods (“Special Character”, “Bolding”, “Italicizing”, and “Brackets”), the ZS-CoT model generally shows higher or comparable performance to the ZS model. The “Brackets” shows a significant improvement when using ZS-CoT, with a jump from 31.05% (ZS) to 36.00% (ZS-CoT). However, the BiasF_{o} metric shows a considerable divergence between 4.89\%^{2} (ZS) and 11.65\%^{2} (ZS-CoT). This depicts that while applying CoT may improve accuracy in certain methods, it does not generally reduce format bias.
Reliability of the results.
Overall, 80\% of the EstTrue results ( 169/210 ) are reliable. Gemma-7B-it shows mixed reliability, with some red-marked scores indicating unreliable results, particularly in the “Quoting” format. This is because Gemma failed to follow the quoting instruction to quote the final answer. Mistral-7B-it exhibits similar variability, with some unreliable scores in “Quoting” and “Placeholder” formats. ChatGPT generally demonstrates mostly reliable results, with only 1 quoting result unreliable.
C.3 List Discussions
We assess Gemma, Mistral, and ChatGPT with two prompting
techniques, Zero-shot (ZS) and Zero-shot Chain-of-Thought (ZS-CoT) ( Section 5.3 ) on two benchmarks SciDocsRR and SemEval2017. Our prompts are provided in E.3 . We utilize FI_{C},SysE,TrueE as our evaluation metrics. The results are illustrated in Tab. 10 . In addition, Tab. 11 , Tab. 12 and Tab. 13 are the distillation results of Tab. 10 :

Tab. 11 . For each model, we average its EstTrueE performance overall benchmarks and prompting techniques. For each task, we average the EstTrueE scores overall models and prompting techniques. This table is drawn in Fig. 4 and its discussions are conducted in Section 5.3 .

Tab. 12 . The purpose of this table is to compare the FI scores across formats. We average all the FI scores across models and tasks.

Tab. 13 . The purpose of this table is to see whether CoT Wei et al. ( 2022 ) mitigates format bias. We average all the EstTrueE scores over all models and benchmarks for each ZS and ZS-CoT prompting method.
Format instruction-following bias.
From Tab. 12 , the “Newline list” format has the highest FI score of 66.05\% , while the FI scores of other formats are around 48.6\% . Among the models, ChatGPT follows the instructions best with FI score 83.27\% . Specifically, ChatGPT excels at following instructions for all formats. In comparison, Mistral performs poorly with “Special character list”, while Gemma struggles the most with “Python” and “Bullet point list”. Explaining these phenomena, we observe that Gemma often misinterprets the task of converting textual samples into lists of numbers as a coding task, resulting in answers that do not match the desired format or content. We take an example in dataset SciDocsRR Section D.1 with detailed prompt as in Section E.3
ZS versus ZS-CoT.
The results, detailed in Tab. 13 indicate that prompting with ZS-CoT substantially enhances model performance across various formats. Moreover, ZS-CoT effectively reduces format bias, as evidenced by the BiasF_{o} metric decreasing from 46.88\%^{2} to 33.69\%^{2} . From this, we conclude that CoT reduces format bias.
Reliability of the results.
From Tab. 10 , 73\%(35/48) of the EstTrue results are reliable. However, some scores of Gemma-7B-it ( 8/16 ) and Mistral-7B-it ( 5/16 ) on these benchmarks are red-marked, indicating unreliable results of this model. In contrast, the ChatGPT’s results are perfectly reliable.
C.4 Mapping Discussions
We select Gemma, Mistral, and ChatGPT for our evaluation, using two prompting techniques: Zero-shot (ZS) and Zero-shot Chain-of-Thought (ZS-CoT) ( Section 5.4 ). These models are tested on the SciREX dataset across three difficulty levels: Easy, Medium, and Hard. Detailed prompt specifications are provided in Section E.4 . We calculate FI_{C},SysE,EstTrueE with the results presented in Tab. 14 . Furthermore, Tab. 15 , Tab. 16 and Tab. 17 are distilled from Tab. 14 :

Tab. 15 . For each model, we average its EstTrueE performance overall benchmarks and prompting techniques. For each task, we average the EstTrueE scores overall models and prompting techniques. This table is numerical results of Fig. 5 and discussed in Section 5.4 .

Tab. 16 . The purpose of this table is to compare the FI scores across formats. We average all the FI scores across models and tasks.

Tab. 17 . The purpose of this table is to see whether CoT Wei et al. ( 2022 ) mitigates format bias. We average all the EstTrueE scores over all models and benchmarks for each ZS and ZS-CoT prompting method.
Format instruction-following bias.
From this Tab. 16 , noticeably, there is a significant difference in the FI scores of two mapping formats (“JSON” and “YAML”). The average FI scores of the “JSON” mapping format is 66.07\% , which is exceptionally higher (double time higher) than the one of the “YAML” mapping format ( 34.62\% ), resulting in high format bias ( BiasF_{o} = 494.64\%^{2} ). Overall, among models, ChatGPT yields the best result of 90.40\% . Mistral fails to follow “YAML” format instruction, e.g. for the question in dataset SciREX Easy Section D.1 with detailed promt as in Section E.4
ZS versus ZS-CoT.
From Tab. 17 , it is evident that the performance of ZS prompting surpasses that of ZS-CoT for both formats. Upon comparing the BiasF_{o} across prompting techniques, we conclude that CoT Wei et al. ( 2022 ) does not mitigate format bias.
Reliability of the results.
From Tab. 14 , 75\% of the EstTrue results are reliable. The reliability of the results in the mapping output format shows variability across different models and formats. Noticeably, “YAML” mapping format results are less reliable than “JSON” ones. On the other hand, ChatGPT illustrates its high reliability in all mapping formats while Mistral-7B-it and Gemma-7B-it are opposite, and all the results in the “YAML” mapping format of these models are unreliable.
C.5 Mitigating Format Bias Results
In this section, we present the numerical results of our proposed techniques for mitigating format biases using ChatGPT on MMLU, as shown in Tab. 18 .

Demonstrations with formats reduce bias (Indexes 7-12). From Tab. 18 indexes 7-12, we observe that using demonstrations with formats generally increases the average of FI scores, from 72.99\% without any demonstration (index 3), to 79.20\% with using one demonstration and 85.32\% with using 5 demonstrations. Moreover, we find that the performance does not scale linearly with the FI score, indicating that simply increasing the FI score does not necessarily improve the models’ performance or reduce format biases.

Repeating format instructions reduces format bias (Indexes 4-6). From Tab. 18 index 6, most of the formats, repeating the format instruction can increase the FI score (compared to index 3), except for the “Placeholder”. Manual investigation reveals that repeatedly using the “Placeholder” format confuses the model about the actual location of the placeholder, leading to the model omitting the format. Nevertheless, this strategy generally reduces the format bias by decreasing the variance of results from formats other than “Placeholder”, leading to overall reduction.

Fine-tuning with additional format data can eliminate format bias (Indexes 13-15). Finetuning mostly eliminates the format bias problem of the LLM with the bias score only 0.71\%^{2} from Tab. 18 indexes 13-15, while increasing the average FI score up to almost perfect with 99.59\% . This demonstrates that finetuning can help LLMs become more familiar with format tokens and requirements, reducing bias towards different formats.
D.1 Dataset Details
We provide descriptions of all datasets we use in this paper.
MMLU Hendrycks et al. ( 2021 ) .
MMLU is a benchmark for evaluating the performance of language models on Multiple Choices Question on a wide range of subjects across STEM, the humanities, social sciences, and other areas, testing the model’s ability to understand and reason in diverse domains.
BBH Suzgun et al. ( 2022 ) .
BBH is a MCQ dataset which includes a variety of challenging benchmarks that require advanced reasoning, comprehension, and other complex cognitive skills.
GSM8K Cobbe et al. ( 2021 ) .
GSM8K is a dataset of 8,000 math word problems designed for grade school students. The problems require not just basic arithmetic but also multi-step reasoning to solve.
HotpotQA Yang et al. ( 2018b ) .
HotpotQA is a question-answering dataset with a focus on multi-hop reasoning. It contains questions that require finding and combining information from multiple Wikipedia articles to derive the answer.
FairytaleQA Xu et al. ( 2022b ) .
FairytaleQA is a dataset designed for evaluating narrative comprehension, particularly in the context of children’s fairytales. It includes questions that test understanding of characters, plots, and settings in fairytales.
SciDocsRR Cohan et al. ( 2020 ) .
SciDocsRR is a dataset for evaluating information retrieval systems, particularly in the scientific domain. It includes tasks like citation prediction, document classification, and other retrieval-based evaluations.
SemEval2017 Augenstein et al. ( 2017b ) .
SemEval2017 is part of an ongoing series of evaluations for semantic analysis in natural language processing. It includes a wide range of tasks such as sentiment analysis, semantic textual similarity, and information extraction.
SciREX Jain et al. ( 2020b ) .
SciREX is a dataset for evaluating models on the task of information extraction from scientific literature. It focuses on extracting entities, relations, and other structured information from research papers.
D.2 Experimental Results
We present the hyperparameters setting for our experiments below.
Gemma-7B-it Team et al. ( 2024 ) .
For Gemma 7B-it, use the weights from Google and Huggingface 2 2 2 https://huggingface.co/google/gemma-7b-it . We use Nucleus Sampling Holtzman et al. ( 2020 ) as our decoding strategy with a p value of 0.95 , a temperature value of 0.1 , and a window size of 1024 .
Mistral-7B-it-v0.2 Jiang et al. ( 2023 ) .
For Mistral 7B-it, use the weights from MistralAI and Huggingface 3 3 3 https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2 . We use Nucleus Sampling Holtzman et al. ( 2020 ) as our decoding strategy with a p value of 0.9 , and a window size of 1024 .
ChatGPT (gpt3.5-turbo-0125) OpenAI ( 2022 ) .
For ChatGPT, we use the system role: “You are helpful assistant!”. We set the “max_tokens” to be 1024, “top_p=1”, “frequency_penalty=0”, “presence_penalty=0”, and the model mode is “gpt3.5-turbo-0125”.
Datasets for finetuning ChatGPT and finetuning setups.
We preprocess the “auxiliary_train” 4 4 4 https://huggingface.co/datasets/cais/mmlu/viewer/auxiliary_train dataset of MMLU Hendrycks et al. ( 2021 ) , resulting in the training set of 6500 samples as discussed in Section 6 . We preprocess a small, distinct validation set with the same ratio as the training set among formats ”20-20-40-40-50-20-50”, resulting in a total of 240 samples for validation.

We use the default finetuning setup of OpenAI for ChatGPT. Our finetuning costs 63.86 US$ .
E.1 MCQ Prompt Details
The input for the models is the combination of the following components:

\textbf{Input}=\textbf{\{non-CoT/CoT $\times$ Char./Text.\} Instruction}+\textbf{Question}+(\textbf{CoT Wrapping})

where non-CoT/CoT Instruction shows that model uses Zero-shot or Chain-of-Thought, given that

non-CoT \times Char. Instruction = “Answer the following multiple-choice question by outputting only the designated character identifier.”

non-CoT \times Text. Instruction = “Answer the following multiple-choice question by outputting the textual value of your choice without the character identifier without any textual description.”

CoT \times Char. Instruction = “Answer the following multiple-choice question step-by-step by outputting only the designated character identifier.”

CoT \times Text. Instruction = “Answer the following multiple-choice question step-by-step by outputting the textual value of your choice without the character identifier.”

Question is the main content of the task and CoT Wrapping is wrapping instruction if using CoT. i.e. CoT Wrapping = “Wrap your final answer by <ANSWER> and </ANSWER>.”
E.2 Wrapping Prompt Details
The input for the models is the combination of the following components:

\textbf{Input}=\textbf{non-CoT/CoT Instruction}+\textbf{Question}+\textbf{Wrapping Format Instruction}

where non-CoT/CoT Instruction shows that model uses Zero-shot or Chain-of-Thought, given that

If MCQ task (MMLU,BBH)

non-CoT Instruction = “Answer the following multiple-choice question by outputting only the designated character identifier.”

CoT Instruction = “Answer the following multiple-choice question step-by-step by outputting only the designated character identifier.”

If generation task (GSM8K, HotpotQA, FairytaleQA)

non-CoT Instruction = “Answer the following question.”

CoT Instruction = “Answer the following question step by step.”

Question is the main content of the task, and Wrapping Format Instruction is the format we want the model to output, detailed as

Special Character wrapping = “Wrap your final answer by <ANSWER> and </ANSWER>.”

Bolding wrapping = “Wrap your final answer in bold by enclosing it with double asterisks.”

Italicizing wrapping = “Wrap your final answer in italics by enclosing it with single asterisks.”

Brackets wrapping = “Wrap your final answer using double square brackets.”

Parentheses wrapping = “Wrap your final answer using double parentheses.”

Placeholder wrapping = “Wrap your final answer by filling in the placeholder below: ‘So the answer is: {{placeholder}}”’

Quoting wrapping = “Wrap your final answer using triple quotation marks.”
E.3 List Prompt Details
For dataset SciDocsRR , the input for the models is the combination of the following components:

\textbf{Input}= \textbf{ Information}+\textbf{Requirement}+\textbf{List Format Instruction} +\textbf{non-CoT / CoT Instruction}+(\textbf{CoT Wrapping})

where

Information = “Given a query, and a list of documents: Topic: Topic. List of documents: Samples ”

Requirement = “You are required to transform the list of documents into a binary list of 1 or 0 where 1 indicates the document brings useful information to the topic, and 0 indicates the document does not bring useful information to the topic.”

List Format Instruction includes four categories:

Python = “Generate your binary list as a Python list”

Bullet = “Generate your binary list using bullet points”

Special Character = “Generate your binary list using <SEP> to separate elements”

New Line = “Generate your binary list such that each element is in a new line”

non-CoT / CoT Instruction includes:

non-CoT Instruction = “without any explanation.”

CoT Instruction = “step by step”

CoT Wrapping = “Wrap your final list by <ANSWER> and </ANSWER> .”

For dataset SemEval2017 , the input for the models is the combination of the following components:

\textbf{Input}= \textbf{ Requirement}+\textbf{Document}+\textbf{List Format Instruction} +\textbf{non-CoT / CoT Instruction}+(\textbf{CoT Wrapping})

where

Requirement = “Extract a list of keyphrases from the following document:”

Document is the main content of the task.

List Format Instruction includes four categories:

Python = “Generate your binary list as a Python list”

Bullet = “Generate your binary list using bullet points”

Special Character = “Generate your binary list using <SEP> to separate elements”

New Line = “Generate your binary list such that each element is in a new line”

non-CoT / CoT Instruction includes:

non-CoT Instruction = “without any explanation.”

CoT Instruction = “step by step”

CoT Wrapping = “Wrap your final list by <ANSWER> and </ANSWER> .”