Venue: NAACL 2025

# LLMs Are Biased Towards Output Formats!

Systematically Evaluating and Mitigating Output Format Bias of LLMs

Do Xuan Long\({}^{1,2}\), Hai Nguyen Ngoc\({}^{3}\), Tiviatis Sim\({}^{1,4}\), Hieu Dao\({}^{1}\), Shafiq Joty\({}^{5,6}\), Kenji Kawaguchi\({}^{1}\), Nancy F. Chen\({}^{2}\), Min-Yen Kan\({}^{1}\)

###### Abstract

We present the first systematic evaluation examining format bias in **performance** of large language models (LLMs). Our approach distinguishes between two categories of an evaluation metric under format constraints to reliably and accurately assess performance: one measures performance when format constraints are adhered to, while the other evaluates performance regardless of constraint adherence. We then define a metric for measuring the format bias of LLMs and establish effective strategies to reduce it. Subsequently, we present our empirical format bias evaluation spanning four commonly used categories--multiple-choice question-answer, wrapping, list, and mapping--covering \(15\) widely-used formats. Our evaluation on eight generation tasks uncovers significant format bias across state-of-the-art LLMs. We further discover that improving the format-instruction following capabilities of LLMs across formats potentially reduces format bias. Based on our evaluation findings, we study prompting and fine-tuning with synthesized format data techniques to mitigate format bias. Our methods successfully reduce the variance in ChatGPT's performance among wrapping formats from \(235.33\) to \(0.71~{}(\%^{2})\).

\begin{tabular}{c} No. \\ \end{tabular} 
\begin{tabular}{c} No.



depicted in Fig. 1. Nonetheless, Case (1) is challenging to automatically measure, requiring costly human investigation. Therefore, we propose a reliable estimator for evaluating LLM performance under format constraints without human intervention by considering format-following scores. We start by redefining LLM evaluation metrics into two distinct classes to construct the estimator, as detailed in SS3.1. Accordingly, we define a metric to quantify format bias in LLMs and establish criteria for evaluating methods that successfully mitigate this bias (SS3.2). Based on these formulations, we present our format evaluation framework, comprising of the widely-utilized categories of multiple-choice question-answer (MCQ; SS5.1), wrapping (SS5.2), list (SS5.3) and mapping formats (SS5.4).

Across \(15\) widely-used formats, our evaluation with zero-shot and zero-shot chain-of-thought prompting Kojima et al. (2022) on eight question-answering and reasoning tasks reveals substantial performance and format-instruction following inequalities. To address this, we examine prompting and fine-tuning using synthesized format data techniques that work for both open- and closed-source LLMs. Our study validates that enhancing LLMs' capabilities to follow format instructions potentially mitigates format bias: (1) Prompting with demonstrations and (2) Repeating format instructions substantially alleviates this bias. Moreover, we investigate (3) Synthesizing limited format data based on our evaluation results for fine-tuning. Our approaches significantly decrease ChatGPT performance variance across wrapping formats from \(235.33\) to \(0.71\) (\(\%^{2}\)) on MMLU Hendrycks et al. (2021). Our key contributions are:

1. We introduce the first systematic framework1 to evaluate format performance bias in LLMs. Footnote 1: Our codes and data will be made publicly available at link.
2. A large-scale evaluation spanning \(15\) formats, \(8\) tasks, and \(4\) models revealing substantial LLM performance variance across formats.
3. The development of \(3\) novel prompting and fine-tuning methods to mitigate this bias.

## 2 Related Works

Large language models (LLMs) have shown remarkable proficiency in formatting outputs to meet human expectations. Such formats include markdown for lists and pointers Achiam et al. (2023), code blocks Gur et al. (2023), and integrate tags, or LaTeX for scientific texts Singh et al. (2023); Wang et al. (2024). Given the rising importance of formatting capabilities in LLMs, recently, format-following benchmarks have been developed for assessing LLMs' adherence to specified formats Zhou et al. (2023); Xia et al. (2024); Chen et al. (2024); Macedo et al. (2024); Liu et al. (2024). However, these studies only evaluate format-instruction following capabilities. _Our research further assesses LLM performance across different formats, uncovering significant format bias in various tasks and models_. We also acknowledge the concurrent work by Tam et al. (2024), which examines the impact of format restrictions on LLM performance. However, unlike our approach, they do not disentangle evaluation metrics under format constraints and only evaluate \(3\) structured formats, substantially fewer than our study.

## 3 Output Format Evaluation Framework

### Theoretical Analysis: Format Evaluation

Automatic evaluation of LLMs in question-answering and reasoning tasks mainly relies on rule-based extraction to identify final answers from generated texts Guo et al. (2023). Within format constraints, determining the model's true performance, which is our focus, can be ambiguous and inaccurate, as correct responses might be overlooked due to format discrepancies (e.g., Case (1) in Fig. 1). To address this, we propose redefining these rule-based evaluation metrics to reliably, transparently and accurately measuring the LLM performance given formats restrictions.

Notations.Suppose that we are interested in evaluating an LLM \(\mathcal{M}\) on a task \(T\) using an evaluation metric \(E\) (such as "Accuracy") under a format constraints \(C\) (such as "Bold the final answer.") on \(n\) samples with the ground-truth answers \(\{y_{1},...,y_{n}\}\) and raw generated answers \(\{\hat{y}_{1},...,\hat{y}_{n}\}\), where \(y_{i},\hat{y}_{i}\in\mathcal{Y}~{}\forall i\) with \(\mathcal{Y}\) being the answer token sequence space. We denote \(F_{C}\) as the binary format-following evaluation function of \(C\):

\[F_{C}(\hat{y}_{i})=\begin{cases}1,&\text{if $\hat{y}_{i}$ satisfies $C$}.\\ 0,&\text{otherwise}.\end{cases} \tag{1}\]

From Eq. (1), we define the Format Instruction-following (FI) Score, denoted as \(FI_{C}\), as the percentage of generated outputs satsifying \(C\) 

\[FI_{C}=\frac{\sum_{i=1}^{n}F_{C}(\hat{y}_{i})}{n}\cdot 100 \tag{2}\]

Prior studies extensively focus on evaluating \(FI_{C}\)Zhou et al. (2023); Xia et al. (2024). Our work further targets evaluating the **performance of LLMs given the format constraints \(C\)**. Under \(C\), we denote \(Ext_{C}()\) as the rule-based answer extractor (or a mixture of extractors) to extract the final answer from \(\hat{y}_{i}\) for comparing it with \(y_{i}\). We define: two evaluation scores based on \(E\):

**Definition III.1** (**Systematic Evaluation Score (\(SysE\))).**

\[SysE=\frac{1}{n}\sum_{i=1}^{n}(E(y_{i},Ext_{C}(\hat{y}_{i})).F_{C}(\hat{y}_{i})) \tag{3}\]

Essentially, \(SysE\) quantifies the performance of \(\mathcal{M}\) on task \(T\) based on the generated answers _that meet the format constraints \(C\)_. For example, in Fig. 1, Case (1) yields a \(SysE\) score of \(0\), while Case (3) achieves \(1\). This also shows that \(SysE\) may not accurately reflect the actual performance of \(\mathcal{M}\) on \(T\), because \(Ext_{C}()\) may fail to extract the final answers from (correct) answers dissatisfying \(C\) (e.g., Case (1) in Fig. 1). We define the True Evaluation Score to address this. Assume that we have an oracle extractor function \(OracExt_{C}()\) that can extract the final answer from \(\hat{y}_{i}\), regardless of whether \(\hat{y}_{i}\) fulfills \(C\), we have:

**Definition III.2** (**True Evaluation Score (\(TrueE\))).**

\[TrueE=\frac{1}{n}\sum_{i=1}^{n}E(y_{i},Oraclext_{C}(\hat{y}_{i})) \tag{4}\]

\(TrueE\) measures the performance of \(\mathcal{M}\) on task \(T\) across all generated answers _given the format constraints \(C\), regardless of format satisfaction_. In Fig. 1, both Cases (1) and (3) achieve a true accuracy of \(1\). This score is crucial for assessing the true performance of LLMs given the format.

Prior studies do not clearly differentiate between \(SysE\) and \(TrueE\). In practice, measuring \(TrueE\) is challenging because \(OracExt_{C}()\) is unavailable. While researchers typically employ a mixture of methods to extract answers, this approach encounters two severe issues. First, these mixture-of-method extractors can be complex, unreliable, and often impractical for large-scale experiments with diverse formats like ours. Second, designing them to be reliable for complex formats such as medical reports can be impossible due to the countless potential errors. Another alternative is to assign a default value to \(Ext_{C}(\hat{y}_{i})\). While this can temporarily avoid cases \(\mathcal{M}\) fails to fulfill \(C\), this is an incorrect practice since the default value may not be the actual output. Reliably measuring \(TrueE\) often requires human investigation Lin et al. (2022) or the fine-tuning of evaluation models as scorers Yang et al. (2024), both of which are costly.

Nevertheless, \(TrueE\) is crucial for a _fair evaluation_ of LLM performance bias across formats. Therefore, we propose a simple estimator of \(TrueE\), denoted as \(EstTrueE\):

\[EstTrueE=\begin{cases}SyEsE.\frac{100}{FI_{C}},&\text{if $FI_{C}\neq 0$}\\ 0,&\text{otherwise}.\end{cases} \tag{5}\]

When \(FI_{C}=0\), estimating \(EstTrueE\) becomes impossible. \(EstTrueE\) enables the fair format bias evaluation because normalizing \(SysE\) by \(FI_{C}\) prevents skewing comparisons of how different formats affect the LLM due to \(FI_{C}\). It is especially useful for large-scale experiments since it is fully automatic. Let the \(EstTrueE\) margin of error be \(\epsilon\) with a confidence interval \(1-\alpha\) and \(S_{C}=n\cdot FI_{C}\) as #generated answers satisfying \(C\).

**Theorem III.1** (**Reliability of \(EstTrueE\)).**\(EstTrueE\) is consistent. Moreover, \(EstTrueE\) is reliable if and only if:

\[FI_{C}\geq\frac{1}{1+n\cdot\left(\frac{\epsilon}{v\cdot s}\right)^{2}} \tag{6}\]

Moreover, we have:

\[\lim_{FI_{C}\to 100}EstTrueE=TrueE \tag{7}\]

where \(s^{2}\) is the sample variance of evaluation scores of generated answers satisfying \(C\) and \(v=t_{\alpha/2,S_{C}-1}\) is the critical value from the t-distribution with \(S_{C}-1\) degrees of freedom.

In summary, we have proposed a consistent estimator \(EstTrueE\) of the true performance of LLMs measured by metric \(E\) under the format constraints (Def. III.2). This estimator is essential because it: (1) ensures transparent and fair LLM performance evaluation across different formats; and (2) supports large-scale format bias evaluation. Note that a high score \(EstTrueE\) is only reliable iff \(FI_{C}\) is high enough (Thm. III.1). Henceforth, unless otherwise specified, \(EstTrueE\) is our primary metric for measuring model performance given format constraints. The proof of Thm. III.1 is in SSB.1 



### Theoretical Analysis: Format Bias

This section defines the metric to quantify format bias and outlines the criteria to mitigate such bias.

Bias measurementTo measure the format bias of the LLM across \(k\) formats \(F_{o}=\{C_{1},\dots,C_{k}\}\), we define a single metric, \(BiasF_{o}\), as the variance of \(EstTrueE\) scores over these \(k\) formats, denoted as \(\{EstTrueE_{1},\dots,EstTrueE_{k}\}\). Let \(\mu_{EstTrueE}=\frac{1}{k}\sum_{i=1}^{k}EstTrueE_{i}\) represent the mean \(EstTrueE\) score. Then:

\[BiasF_{o}=\frac{1}{k}\sum_{i=1}^{k}(EstTrueE_{i}-\mu_{EstTrueE})^{2} \tag{8}\]

Realiability of \(BiasF_{o}\).By Eq. (8), the lower \(BiasF_{o}\) is, the less format-\(F_{o}\)-biased \(\mathcal{M}\) is, suggesting a criterion for mitigating output format bias. However, \(BiasF_{o}\) is an estimator based on the estimators \(EstTrueE_{i}\). Therefore, to enhance the reliability of \(BiasF_{o}\), it is also necessary to improve the reliability of \(EstTrueE_{i}\) by increasing \(FI_{C_{i}}~{}\forall i\) (Thm. 3.1). Therefore, we propose **two necessary criteria for an effective method to mitigate format bias in LLMs**: **(i) Minimize bias metric:** reducing \(BiasF_{o}\), indicating less format-\(F_{o}\)-bias in \(\mathcal{M}\); **(ii) Increase the format-following scores for all formats:** ensuring the reliability of \(BiasF_{o}\) by increasing the \(FI\) scores across all the formats: \(\{FI_{C_{1}},...,FI_{C_{k}}\}\) (Eq. (2)).

### Formats for Evaluation

We establish \(4\) format categories for evaluation consisting of \(15\) formats introduced by prior practices:

(i) Multiple-choice question (MCQ) answer (SS5.1).where LLMs answer questions by selecting from provided choices, presented as either a **(1) Character identifier**Robinson and Wingate (2023); or **(2) Choice value**Chen et al. (2023).

(ii) Wrapping (SS5.2).where LLMs must enclose the final answer within the two characters, which is crucial for automatic evaluation to isolate the final answer from reasoning thoughts. We focus on evaluating \(7\) widely used wrapping strategies: **(1) Special character**Gur et al. (2023); **(2) Bolding**Zhou et al. (2023); **(3) Italicizing**Zhou et al. (2023); **(4) Double brackets**Luo et al. (2024); **(5) Double parentheses; **(6) Placeholder**Wang et al. (2024); **(7) Quoting**Zhou et al. (2023).

(iii) List (SS5.3).where the output of LLMs is a list of elements. We investigate \(4\) formats representing lists: **(1) Python list**Do et al. (2025); **(2) Bullet-point list**Liu et al. (2024); **(3) List of elements separated by a special character "[SEP]*Boucher (2023); and **(4) List of elements arranged on separate lines**Mishra (2023).

(iv) Mapping (SS5.4).where LLMs are employed to output dictionaries or maps. We focus on two ubiquitously used mapping structures: **(1) Python dictionary/JSON**JavaScript Object Notation)Baumann et al. (2024) and **(2) YAML**Yet Another Markup Language)Goel et al. (2023).

Format-instruction following.We introduce Appx.-Alg. 1, a rule-based heuristic to determine the format-instruction following function \(F_{C}\) (Eq. (1)) for our benchmarked formats. It calculates the binary FI score by verifying that the generated output includes the specified formatting tokens and that the extracted final answer matches the expected type. It is highly extendable to other formats ( SSA).

## 4 General Experimental Setups

Benchmarks.For MCQ bias evaluation (SS5.1), we select two datasets: **MMLU**Hendrycks et al. (2021) and **BBH**Suzgun et al. (2023). For MMLU, we randomly choose 27 subcategories. For BBH, we select the sports_understanding category following Gupta et al. (2024). For wrapping bias assessment (SS5.2), in addition to MCQ benchmarks, the following datasets are experimented: ** GSM8K**Cobbe et al. (2021) for reasoning, **FairstaleQA**Xu et al. (2022) for narrative comprehension, and **HotpotQA**Yang et al. (2018) for multi-hop reasoning. For list bias investigation (SS5.3), we use **SciDocSRR**Muennighoff et al. (2023), a scientific document ranking task as the order list generation task, and **SemEval 2017**Augenstein et al. (2017), the keyphrase extraction task as the unordered list generation. For mapping bias examination (SS5.4), we utilize a document-level information extraction task named **SciREX**Jain et al. (2020) by synthesizing three extraction difficulty levels: easy (extracting from \(1\) sentence for \(1\) category), medium (\(3\) sentences, \(2\) categories), and hard (\(5\) sentences, \(4\) categories). For all benchmarks except MCQ, we sample \(200\) points for evaluation Bai et al. (2024).

Models.We select both open- and closed-source LLMs for our evaluation: **GEMma-7B-it**Team et al. (2024), **Mistral-7B-it-v0.2**Jiang et al. (

2023), and **Llama-3.1-8B-it**(Dubey et al., 2024) for open-source as they are among state-of-the-art open-source LLMs; **ChatGPT (gpt-3.5-turbo-0125)** for closed-source as this premier chatbot possesses superior instruction-following ability. Our purpose is not to reproduce the models' performance, but to show the bias.

Metrics.Following our discussion in SS3.1, we disentangle **Accuracy (Acc)** for MMLU and BBH Guo et al. (2023); **F1** for GSM8K, HotpotQA, FairytaleQA; and **Mean Average Precision (MAP)** for SciDocsRR Muennighoff et al. (2023) and we report the metrics \(EstTrueAcc\), \(EstTrueF1\), \(EstTrueMAP\) (Eq. (5)) in the main text. For metrics' reliability, we set \(\alpha=\epsilon=5\%\).

Prompting baselines.Our focus is on two widely used prompting baselines: (1) **Zero-shot (ZS)** prompting and (2) **Zero-shot Chain-of-Thought (ZS-CoT)** prompting Kojima et al. (2022). For the ZS baseline, we instruct LLMs to answer the question with the prompt "Answer the following question..." followed by the suffix "without any explanation". For ZS-CoT, we use the suffix "step-by-step" instead. For the ZS-CoT experiments in Sections 5.1, 5.3 and 5.4, LLMs are instructed to wrap the final answer by "<ANSWER>" and "</ANSWER>" tokens to distinctly isolate it from the reasoning chains (see Tab. 1 for the wrapping instruction). We use this wrapping method since our experiment in SS5.2 shows that it achieves the highest instruction-following score on average across LLMs. Detailed prompts are provided in SSE. We average the performance under two prompting methods to report in the main text.

## 5 Format Evaluation Experiments

Overall, we find that: (1) Models show substantial format-following bias across formats for all benchmarks; (2) For all models and datasets, significant performance bias exists across formats; (3) \(78.30\%\) of the \(EstTrue\) results are reliable (with \(70\%\) for MCQ, \(82.5\%\) for wrapping, \(67.19\%\) for list, and \(77.08\%\) for mapping formats) _highlighting significant weaknesses of LLMs in following format instructions_. We dive into (2) for every format as it is our main focus, (1, 3) are discussed in detail in Appendices C.1 to C.4.

### Experiments on MCQ Format

Setups.We investigate the bias of LLMs towards different MCQ output formats. We assess two formats as introduced in SS3.3: (1) Character identifier and (2) Choice value. For example, if the choice is "[A. Yes, B. No]", then the character identifier can be "A/B", while the choice value can be "Yes/No". We exclude the format combining the character identifier and choice value (such as "A. Yes") from our evaluation because instructing LLMs to output this format can be non-trivial and require manual effort to craft instructions tailored for different models. To ensure that LLMs understand the "Character identifier" and "Choice value" as we expect, we add a contrastive format requirement to the prompts (e.g., "without any textual description" for the "Character identifier" prompts).

Results.Fig. 2 provides a synopsis of our evaluation results, with numerical values shown in Appx.-Tab. 2. From Fig. 2-left, we observe that Mistral possesses the highest disparity between the two MCQ answer formats, with \(58.69\%\) accuracy on average for character and only \(4.22\%\) for textual value. Additionally, despite ChatGPT often being regarded as one of the most robust LLMs, it shows a significant performance difference between the two formats (\(19.03\%\)). Overall, LLMs are heavily biased towards outputting character identifiers. Requiring them to generate the choice's value causes notable performance drops on most models.

From Fig. 2-right, we notice that the models exhibit higher bias on BBH, which appears to be an easier benchmark than MMLU. We attribute this to the small size of BBH, which makes the performance more sensitive to format variations.

Why such bias?We hypothesize the root cause of the significant performance bias across different formats is the **format token bias** of LLMs. The

Figure 2: Average estimated true accuracy (§3.1) results of MCQ benchmarks across models (left) and datasets (right) showing performance bias of LLMs across formats.

 

non-uniform distribution of FI scores among formats suggests that the models assign probabilities to format instructions differently based on their training data. This leads to varying prior assignments of probabilities to specific tokens, causing final predictions non-uniformly distributed across formats. This hypothesis is supported by our simple fine-tuning with formatted data, which familiarizes LLMs with format instructions relatively equally leading to a drastic format bias reduction (SS6). This emphasizes the necessity of more research in fine-tuning LLMs to reduce format bias and raises concerns about the reliability and reproducibility of recent studies using varied formats.

### Experiments on Wrapping Format

Setups.We study LLM bias towards \(7\) wrapping methods: (1) Special character; (2) Bolding; (3) Italicizing; (4) Brackets; (5) Parentheses; (6) Placeholder; (7) Quoting, detailed in Tab. 1. We evaluate LLM performance across formats on the MMLU, BBH, GSM8K, FairytaleQA, and HotpotQA.

Results.Fig. 3 outlines an overview of our evaluation outcomes with results in Appx.-Tab. 6. From Fig. 3-left, we see that Llama exhibits the highest bias towards different formats (with a \(BiasF_{o}\) value of \(74.86\%^{2}\); see Appx.-Tab. 7), while ChatGPT performs the best. Notably, for "Quoting" and "Parenthesis", the Gemma follows instructions only about \(0-4\%\) yielding nearly zero performance, highlighting its critical weaknesses. Among the 7 formats, "Placeholder" (\(37.15\%\)) proves to be the most effective wrapping output format, while "Quoting" (\(24.58\%\)), "Parenthesis" (\(28.57\%\)) are among those that achieve the lowest performance.

From Fig. 3-right, models exhibit bias across all tasks, with the lowest on MMLU (\(16.58\%^{2}\); see Appx.-Tab. 7) possibly because the models already performed relatively well on it, and the highest on BBH (\(54.58\%^{2}\)), the challenging task without train data. This demonstrates the pervasive presence of wrapping bias in LLMs.

Why such bias?The format token bias of LLMs as explained in SS5.1 is also our hypothesis. Specifically, we found the low performance of the "Quoting" and "Parenthesis" because, in generation tasks, models often wrap (via quoting/parenthesizing) not only the final answer, as instructed, but also parts of the context (e.g., ""The answer is 3."), leading to poor F1 scores. Moreover, Gemma completely ignores the above format instructions, resulting in 0% FI scores, which also contribute to the low average estimated F1 scores. These strongly indicate the presence of format token bias.

### Experiments on List Format

Setups.We explore the bias of LLMs in generating lists following \(4\) formats: (1) Python list, (2) Bullet-point list, (3) Character-separated list, and (4) Newline-separated list. We evaluate the models on two list generation tasks: _(i) Unordered list_, using the keyphrase extraction task on the SemEval 2017 dataset, and _(ii) Ordered list_, using the document ranking problem on the SciDocsRR task.

Results.Fig. 4 displays the key findings of our evaluation across models and datasets with numerical results in Appx.-Tab. 10. From Fig. 4-left, we notice that Mistral exhibits the most bias, with the \(BiasF_{o}\) value of \(353.80\%^{2}\). In contrast, ChatGPT and Gemma show much lower bias, with values of \(7.08\%^{2}\) and \(1.32\%^{2}\), respectively. Of the four formats, the "Python" and "Newline-separated' formats yield the highest performance, likely due to models trained extensively on code data. Conversely, the "Bullet-point list" format results in the lowest performance, particularly for Mistral, highlighting the inherent bias for such formats.

The performance bias is regardless of the task as plotted in Fig. 4-right, with the highest \(BiasF_{o}\) value of \(54.12\%^{2}\) on the order list generation task SciDocsRR, and significantly lower (\(31.86\%^{2}\)) on SemEval2017 task. The high bias in the SciDocsRR task is because Mistral and Gemma mostly failed to perform this task following the "Bullet" and "Special character" list formats while excelling in solving it following the other formats.

Why such bias?We attribute the bias to the format token bias (SS5.1). Since the models were extensively trained on code data, they excel in solving code-related instructions. In contrast, "Bullett-point" and "Special character" lists are much less common. One interesting case is Gemma where it

\begin{table}
\begin{tabular}{|l|c|l|} \hline \multicolumn{1}{||c|}{**Wrapping type**} & \multicolumn{1}{c|}{**(start, end)**} & \multicolumn{1}{c|}{**Prangeit View your fast assurer..**} \\ \hline Spectat. & (CANNEPQ-GANNEPQ) & 9 vANSERQ-GANNEPQ- \\ \hline Budging & (*+,”) & \# healthy cognitive path could networks. \\ \hline Indicative & (*,”) & \# a table containing with single asterisks. \\ \hline Backtrack & (II, II) & setup double-aware brackets. \\ \hline Pennebises & (III, III) & using double connections. \\ \hline Purchadder & Nest & \# filling into the default below: \\  & & ”Sho the answer is: plastichediff” \\ \hline Queing & (“,”, “) & using triple double-operation marks. \\ \hline \end{tabular}
\end{table}
Table 1: Wrapping “start” and “end” tokens with instructions.

 

performed worse on generating "Python" lists compared to "Bullet-point" lists. Our analysis suggests that Gemma misinterprets the format instruction as a coding request, generating Python code programs instead of an answer in a Python list, suggesting Gemma was predominantly trained on code data.

### Experiments on Mapping Format

**Setups.** We examine the performance bias of LLMs on two mapping formats as discussed in SS3: (1) Python dictionary/JSON; (2) YAML. We preprocess the SciREX task Jain et al. (2020) as described in SS4 into three extraction levels: (i) Easy (1 sentence, "Task" category); (2) Medium (3 sentences, "Task, Method"); (3) Hard (5 sentences, "Task, Method, Material, Metric" categories).

Results.Fig. 5 illustrates a summary of our evaluation with numerical details in Appdx.-Tab. 14. From Fig. 5-left, Gemma is the most biased, with a performance gap of \(16.51\%\) between the two formats, followed by Mistral with a \(16.07\%\) gap. ChatGPT and Llama, however, are relatively robust against format variations. On average, JSON performs significantly better than YAML for mapping, likely because more JSON data is used to train models due to its popularity.

From Fig. 5-right, extracting \(4\) categories in the Hard task shows the largest performance gap between mapping formats. Surprisingly, the Medium task displays the least bias, likely because models perform best in this task.

Why such bias?The bias is attributed to the format token bias (SS5.1). While Mistral excels in generating JSON, it and Gemma struggle with YAML. Even successfully generating YAML output, Mistral and Gemma frequently introduce noisy information (88%-65% for Mistral with and without CoT, 98%-79% for Gamma) in the response (e.g., a key "Task" should have multiple values, Mistral generates multiple key-value pairs instead e.g., "Task_1:Training ... Task_2: ..."), resulting in poor overall performance.

Figure 4: Average \(EstTrueF1\) (SemEval2017) and \(EstTrueMAP\) (SciDocsRR) (§3.1) across models (left) and benchmarks (right) showing performance difference of LLMs across \(4\) widely used list formats.

Figure 5: Average estimated true F1 scores (§3.1) across models (left) and benchmarks (right) showing performance bias of LLMs across \(2\) widely used mapping formats.

Figure 3: Average estimated true Accuracy (MCQ) and F1 (GSM8K, HotpotQA, FairtaleQA) scores (§3.1) across models (left) and across benchmarks (right), showing performance bias of LLMs across \(7\) widely used wrapping methods.

 



## 6 Mitigating Performance Format Bias: Actionable Recommendations

We propose methods as actionable recommendations to mitigate format bias in the performance of LLMs. Generally, three primary streams of techniques have been widely studied and applied to tackle LM biases: (1) Prompting Xu et al. (2024); Macedo et al. (2024); (2) Calibrating Roelofs et al. (2022); Li et al. (2024); and (3) Fine-tuning Schick et al. (2021); Ghaddar et al. (2021). While calibration techniques can only be used for white-box models, prompting and fine-tuning can be applied for both black-box (via API) and white-box ones. Therefore, we explore prompting and fine-tuning techniques to reduce format bias. We target mitigating the format bias of **ChatGPT** in Fig. 6, the strongest model that we benchmarked, on **MMLU**. We aim to reduce the **wrapping** bias (SS5.2) due to resource limits, but our methods can be generalized to any model and format. We also verify (and confirm) our mitigation strategies on **Gamma-2B-it**, a medium-size model in Appx.-Tab. 18.

Demonstration(s) reduce(s) format bias.As discussed in SS5.1, LLMs show bias across formats possibly because of the token bias issue, causing LLMs to non-uniformly comprehend the format instructions. To address this, we examine whether demonstrations with formats can reduce such bias, as they are commonly utilized to enhance LLM's comprehension of the task patterns Xie et al. (2022). Particularly, for each wrapping format in SS5.2, we select \(1\) and \(5\) random samples from the auxiliary train data of MMLU and manually format the answers as demonstrations. The results are outlined in Fig. 6. Firstly, incorporating demonstrations typically enhances the FI scores of the model (from 72.99% to 79.19% and 85.31%) (i), with five demonstrations yielding the most. Secondly, we observe a notable decrease in the \({BiasF_{o}}\) score (ii) upon supplementing demonstrations. From (i), (ii) and SS3.2, we conclude integrating demonstrations mitigates format bias.

Repeating format instructions reduces format bias.We found that repeating instructions generally increases FI scores (i) across most formats except "Placeholder", which can consequently lessen the mode's token bias towards format instructions (SS5.1). Using our two proposed criteria for effective format bias mitigation in SS3.2, it is worth examining if this approach reduces \({BiasF_{o}}\), thereby being an effective mitigation. Our answer is yes. By repeating the wrapping instructions of ChatGPT thrice, we observed a decrease in the \({BiasF_{o}}\) (ii) score presented in Fig. 6. Combining (i) and (ii) suggests that this strategy is an effective mitigation. For "Placeholder," human investigation reveals that multiple placeholder instructions cause ChatGPT to be confused about where the placeholder is, making it frequently misunderstand and fail to follow this format instruction.

Fine-tuning with additional format data can eliminate format bias.We hypothesize that completely solving the format token bias problem of LLMs necessitates finetuning them on format data so that they are familiar with tokens in format instructions evenly. We propose a simple data synthesis strategy for finetuning LLMs: we sample a small set of training data for all evaluated formats, with ratios inversely proportional to their **systematic evaluation scores** (SS3.1). We chose \(SysE\) scores over the \(EstTrueE\) because they reflect the current model performance. Practically, based on ChatGPT's zero-shot systematic performance on MMLU colored in blue in Appx.-Tab. 6, we approximate the formats' performance ratios as "\(1,1,\frac{1}{2},\frac{1}{2},\frac{1}{3},1,\frac{1}{3}\)" from left-to-right, resulting in training data ratios of formats of "\(1,1,2,2,3,1,3\)". We then preprocess the MMLU auxiliary training data according to these ratios, scaled by \(500\) (\(6500\) samples total), and train ChatGPT on this dataset. The finetuned results are plotted in Fig. 6. Firstly, after finetuning, the average FI score across all formats is nearly perfect at \(99.58\%\) (ii). Secondly, the \({BiasF_{o}}\) score is significantly reduced from \(235.33\%^{2}\) to \(0.71\%^{2}\) (ii). These (i) and (ii) indicate finetuning largely eliminates format bias.

Figure 6: More demonstrations and repeating format instructions mitigate format bias. Finetuning mostly eliminates the format bias. The performance is reported using ChatGPT on MMLU (Appx.-Tab. 18 for num. results).



Conclusions

We introduce the pioneering systematic investigation of format bias in LLM performance, revealing significant biases across widely used formats for all models and benchmarks. Our method involves developing metrics to assess this bias and establishing criteria for effective mitigation. We then introduce prompting and fine-tuning techniques to alleviate format bias based on our evaluation findings. Our work aims to sharpen the focus of future LLM research toward fairer and more robust development.

### Limitations

Our study has several limitations. Firstly, the metrics \(EstTrue\) and \(BiasF_{o}\) proposed in SS3.1 and SS3.2 are estimators, not exact measures. As discussed, determining \(TrueE\) (Eq. (4)) is infeasible, especially for large-scale experiments across various models and datasets. Achieving this would require extensive fine-tuning and comprehensive human evaluations, both prohibitively expensive and impractical in many scenarios. Our proposed metrics \(EstTrue\) and \(BiasF_{o}\) are handy for large-scale experiments with multiple models and datasets due to their fully automatic nature. We further propose Thm. 3.1 to validate the reliability of \(TrueE\) statistically.

Secondly, our empirical evaluation of format bias is limited by computational and budget constraints to specific datasets, formats, and models. This restriction limits the generalizability of our findings and may obscure further insights that could be gained from expanding the experiments to include more formats, larger-scale datasets, and additional task categories.

Finally, while our study primarily attributes format bias to token bias in the training data of LLMs and proposes data-focused approach, it does not extensively explore other factors related to model architecture and training processes. This omission represents a significant area for future research, as more fundamental, architecture-level solutions could be crucial, for addressing format bias in LLMs. Our study underscores the importance of continued research dedicated to quantifying and mitigating format bias.

## Ethical Considerations

Our work uncovers significant format bias in LLMs, raising concerns regarding fairness and potential discrimination in real-world applications.

Bias and fairness.Format bias in LLMs can result in unfair treatment, especially in tasks where multiple possible formats can be used. Our research suggests ways to identify and mitigate format bias, aiming for fairer and more equitable LLM applications.

Societal impact.Format bias in LLMs has the potential to disproportionately impact specific populations, as different demographics may have preferences for different communication formats. Further research is essential to fully understand its societal implications and ensure fairness across diverse demographics.

## Acknowledgements

This research project is partially supported by the National Research Foundation Singapore under the AI Singapore Programme (AISG Award No: AISG2-TC-2023-010-SGIL), the Singapore Ministry of Education Academic Research Fund Tier 1 (Award No: T1 251RES2207), and the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-GC-2022-005). DXL and TS are supported by the A*STAR Computing and Information Science (ACIS) scholarship. We thank Goh Yisheng for his contribution in the initial stage of the project. We thank members of WING and Deep Learning Lab, NUS, as well as the anonymous reviewers for the constructive feedback.

## References

* Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Qtpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: SS1.
* Amemiya (1985)Advanced econometrics. Harvard university press. External Links: Link Cited by: SS6.
* extracting keyphrases and relations from scientific publications. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), Vancouver, Canada, pp. 546-555. External Links: Link Cited by: SS6.
* Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, Y. Dong, J. Tang, and J. Li (2024)LongBench: a bilingual, multitask benchmark for long context understanding. In 

Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, pp. 3119-3137. External Links: Link Cited by: SS4.
* N. Baumann, A. Brinkmann, and C. Bizer (2024)Using lms for the extraction and normalization of product attribute values. arXiv preprint arXiv:2403.02130. Cited by: SS5.
* A. Boucher (2023)LIM based context splitter for large documents. External Links: Link Cited by: SS5.
* T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D Kaplan, P. Dhariwal, Arvind Neelakantan, P. Shyam, G. Sastry, A. Asskell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henghian, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei (2020)Language models are few-shot learners. In Advances in Neural Information Processing Systems, Vol. 33, pp. 1877-1901. External Links: Link Cited by: SS5.
* Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang, et al. (2024)A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology15 (3), pp. 1-45. External Links: ISSN 1573-030X, Document, Link Cited by: SS5.
* X. Chen, R. Aksitov, U. Alon, J. Ren, K. Xiao, P. Yin, Sushant Prakash, C. Sutton, X. Wang, and D. Zhou (2023)Universal self-consistency for large language model generation.. arXiv preprint arXiv:2311.17311. Cited by: SS5.
* Y. Chen, B. Xu, Q. Wang, Y. Liu, and Z. Mao (2024)Benchmarking large language models on controllable generation under diversified instructions. arXiv preprint arXiv:2401.00690. Cited by: SS5.
* L. Cheng, X. Li, and L. Bing (2023)Is GPT-4 a good data analyst?. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, pp. 9496-9514. External Links: Link Cited by: SS5.
* C. Chiang and H. Lee (2023)Can large language models be an alternative to human evaluations?. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 15607-15631. External Links: Link Cited by: SS5.
* A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. M. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. C. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-Ari, P. Yin, T. Duke, A. Leskaya, S. Ghemawat, S. Dew, H. Michaelswi, X. Garcia, V. Misra, K. Robinson, L. Fedus, D. Zhou, D. Ippolito, D. Luan, H. Lim, B. Zoph, A. Spiridonov, R. Sepassi, D. Dohan, S. Agrawal, M. Omernick, A. M. Dai, T. Sankaranarayana Pilali, M. Pellat, A. Lewkowyczic, E. Moreira, R. Child, O. Polozow, K. Lee, Z. Zhou, X. Wang, B. Saeta, M. Diaz, O. Firat, M. Catasta, J. Wei, K. S. Meier-Hellsten, D. Eck, J. Dean, S. Petrov, and N. Fiedel (2022)Palm: scaling language modeling with pathways. J. Mach. Learn. Res.24 (1), pp. 240:113. External Links: Link Cited by: SS5.
* J. Clusmann, F. R. Kolbinger, H. Sophie Muti, Z. I Carrero, J. Eckardt, N. Ghaffari Lahel, C. Maria Lavinia Loffler, S. Schwarzkopf, M. Unger, G. P. Velthuzen, et al. (2023)The future landscape of large language models in medicine. Communications medicine3 (1), pp. 141. External Links: Link Cited by: SS5.
* K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. (2021)Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: SS5.
* A. Cohan, S. Feldman, I. Beltagy, D. Downey, and D. Weld (2020)SPECTER: document-level representation learning using citation-informed transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 2270-2282. External Links: Link Cited by: SS5.
* X. Dong, K. Kawaguchi, M. Kan, and N. Chen (2025)Aligning large language models with human opinions through persona selection and value-belief-norm reasoning. In Proceedings of the 31st International Conference on Computational Linguistics, Abuhataju, India, pp. 2526-2547. External Links: Link Cited by: SS5.
* A. Dubey, A. Jauhr, A. Pandey, A. Kadian, A. Al-Dahale, A. Letman, A. Mathur, A. Schelten, A. Yamg, A. Fan, A. Goyal, A. S. Hartshorn, A. Yang, A. Mitra, A. Szrauchumgar, A. Kronev, A. Hinnsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Brion, B. Tang, B. Chern, C. Caucheteux, C. Niayak, C. Bi, C. Marra, C. McConnell, C. Keller, C. Touret, C. Wu, C. Wong, C. Canton Ferrer, C. Nikoladis, D. Allousius, D. Song, D. Pintz, D. Livshits, D. Esioub, D. Choudhary, D. Mahajan, D. Garcia-Olano, D. Perino, D. Hupkes, E. Lakomkin, E. A. AlbBadawy, E. Lobanova, E. Dinan, E. Michael Smith, F. Radenovic, 

Frank Zhang, Gabriele Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Gregoire Mialon, Guangdong Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Laurens Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnston, Joshua Saxe, Ju-Qing Jia, Kalyan Vasuden Alwala, K Upasani, Kate Plawiak, Keqian Li, Ken-591 neth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Khitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala-Yeary, Laures van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madau, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline C. Muzzi, Mahesh Babu Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melissa Hall Melanie Kambadur, Mike Lewis, Min Si, Miets Kumar Singh, Mona Hassan, Namam Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri S. Chattererji, Olivier Duchenne, Onur c Celebi, Patrick Alrarssy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Purin Xu, Qing He, Qingsxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silverius Cabral, Robert Stoijnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabaspapa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Chandra Barpathy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Sciolom, Tobias Speckbacher, Teodor Mihalyo, Tong Xiao, Ujjwal Karn, Vedanuj Gosowami, Viibor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonuguet, Virginie Do, Visb Vogci, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyn Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yiqian Wen, Yuwen Song, Yuchang Zhang, Yue Li, Unning Mao, Zacharie Delpiere Coudert, Zhengxu Yan, Zhengxiing Chen, Zoe Papaxipos, Aaditya K. Singh, Aaron Grattafori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adi Gangidi, Adolfo Victoria, Ahura Golstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Beewski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashw Bharame, Assif Eisennam, Azadeh Yazdan, Beau James, Ben Maurer, Ben Leonhardi, Bernie Huang, Beth Boyd, Betto De Paola, Bhargavi Parajapeape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changshan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beatty, Daniel Kreymer, Shang-Wen Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcante, Evan Dunbar, Evan Monthers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggini, Francisco Guzm'an, Frank J. Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory G. Sizov, Guangyi Zhang, Guna Lakshminarayan, Hamid Shojanazeri, Han Zou, Hannah Wang, Han Zhao, Hawrion Rudolph, Helen Suk, Henry Asepegren, Hunter Goldman, Igor Molbyog, Igor Tufunaro, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhiet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zeremy Reizeenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carrill, Jon Sherpard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kaixing(Kai) Wu, U KamHou, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelea, Keqian Li, Kun Huang, Kulnal Chawka, Kushal Lakhotia, Kyle Huang, Lailin Chen, Laskaya Garg, A Lavender, Lenandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Timpoulkelli, Martynas Markus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghen Kenelly, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanet, Mohammad Rastegari, Munishi Bansal, Nandhnii Smithanan, Natascha Parks, Natasha White, Navyata Baswa, Nayan Singhal, Nick Eyebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chermoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Pontrager, Pierre Roux, Piotr Dollar, Polina Zvyagin, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Rabothran Murthy, Raghu Navani, Rahult Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswarni, Russ Howes, Ruty Binott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun 

Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharndan Ramsawamy, Shaun Lindsay, Sheng Feng, Shengao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigibe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sung-Bae Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tezook Shaked, Varun Vontinituta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Andrei Poenaru, Vlad T. Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xia Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xie, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu Wang, Yuchen Hao, Yunqi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Roshbrick, Zhadouo Wen, Zhenyu Yang, and Zhiwei Zhao. 2024. The llama 3 herd of models. _arXiv preprint_, arXiv:2407.21783.
* Ghaddar et al. (2021) Abbas Ghaddar, Phillippe Langlais, Mehdi Rezagholizabeh, and Ahmad Rashid. 2021. End-to-end self-debiasing framework for robust NLU training. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 1923-1929, Online. Association for Computational Linguistics.
* Goel et al. (2023) Akshay Goel, Almog Gueta, Omry Gilon, Chang Liu, Sofia Erell, Lan Huong Nguyen, Xiaohong Hao, Bolous Jaber, Shashir Reddy, Rupesh Kartha, Jean Steiner, Itay Laishi, and Amir Feder. 2023. LLms accelerate annotation for medical information extraction. In _Proceedings of the 3rd Machine Learning for Health Symposium_, volume 225 of _Proceedings of Machine Learning Research_, pages 82-100. PMLR.
* Guo et al. (2023) Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong, et al. 2023. Evaluating large language models: A comprehensive survey. _arXiv preprint arXiv:2310.19736_.
* Gupta et al. (2024) Shashank Gupta, Vaishnavi Shrivastava, Ameet Deshpande, Ashwin Kalyan, Peter Clark, Ashish Sabharwal, and Tushar Khot. 2024. Bias runs deep: Implicit reasoning biases in persona-assigned LLMs. In _The Twelfth International Conference on Learning Representations_.
* Gur et al. (2023) Izzeddin Gur, Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin Huang, Akankssha Chowdhery, Sharan Narang, Noah Fiedel, and Aleksandra Faust. 2023. Understanding HTML with large language models. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 2803-2821, Singapore. Association for Computational Linguistics.
* Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In _International Conference on Learning Representations_.
* Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In _International Conference on Learning Representations_.
* Jain et al. (2020) Sarthak Jain, Madeleine van Zuylen, Hannaneh Hajishirzi, and Iz Beltagy. 2020. SciREX: A challenge dataset for document-level information extraction. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7506-7516, Online. Association for Computational Linguistics.
* Jian et al. (2020) Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianana Lengyel, Guillaume Lample, Lucile Saulnier, L'eliio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothee Lacroix, and William El Sayed. 2023. Msiral 7b. _arXiv preprint arXiv:2310.06825_.
* Kojima et al. (2022) Takashi Kojima, Shixiang Shane Gu, Michel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In _Advances in Neural Information Processing Systems_.
* Li et al. (2024) Ang Li, Jingqian Zhao, Bin Liang, Lin Gui, Hui Wang, Xi Zeng, Kam-Fai Wong, and Ruifeng Xu. 2024. Mitigating biases of large language models in stance detection with calibration. _arXiv preprint arXiv:2402.14296_.
* Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 3214-3252, Dublin, Ireland. Association for Computational Linguistics.
* Liu et al. (2024) Michael Xieyang Liu, Frederick Liu, Alexander J Flannaca, Terry Koo, Lucas Dixon, Michael Terry, and Carrie J Cai. 2024. "we need structured output": Towards user-centered constraints on large language model output. In _Extended Abstracts of the CHI Conference on Human Factors in Computing Systems_, pages 1-9.
* Liu et al. (2023) Shang-Ching Liu, ShengKun Wang, Tsungyao Chang, Wenqi Lin, Chung-Wei Hsiumg, Yi-Chen Hsieh, YuPing Cheng, Sian-Hong Luo, and Jianwei Zhang. 2023. JarvIX: A LLM no code platform for tabular data analysis and optimization. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track_, pages 622-630, Singapore. Association for Computational Linguistics.
* Luo et al. (2021) Xiaoliang Luo, Akilles Rechardt, GuangZhi Sun, Kevin K. Nejad, Felipe Yanez, Bati Yilmaz, Kangjooe Lee, Alexandra O. Cohen, Valentina Borghesani, Anton Pashkov, Daniele Marinazzo, Jonathan Nicholas, 

Alessandro Salatiello, Ilia Suchditsky, Pasquale Minervini, Sepbera Razavi, Roberta Rocca, Elkhan Yusifo, Tereza Okalova, Nianlong Gu, Martin Feriac, Mikail Khona, Kaustubh R. Patil, Pui-Shee Lee, Rui Mata, Nicholas E. Myers, Jennifer K Bizley, Sebastian Musslick, Isil Poyraz Bilgin, Guiomar Niso, Justin M. Ales, Michael Gaebler, N Apurva Ratan Murty, Leyla Loued-Khenissi, Anna Behler, Chloe M. Hall, Jessica Dafflon, Sherry Dongqi Bao, and Bradley C. Love. 2024. Large language models surpass human experts in predicting neuroscience results. _arXiv preprint arXiv:2403.03230_.
* Macedo et al. (2024) Marcos Macedo, Yuan Tian, Filipe R Congo, and Bram Adams. 2024. Exploring the impact of the output format on the evaluation of large language models for code translation. _arXiv preprint arXiv:2403.17214_.
* Mishra (2023) Onkar Mishra. 2023. Using langchain for question answering on own data.
* Muennighoff et al. (2023) Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB: Massive text embedding benchmark. In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_, pages 2014-2037, Dubrovnik, Croatia. Association for Computational Linguistics.
* OpenAI (2022) OpenAI. 2022. Introducing chatgpt.
* Robinson and Wingate (2023) Joshua Robinson and David Wingate. 2023. Leveraging large language models for multiple choice question answering. In _The Eleventh International Conference on Learning Representations_.
* Roelofs et al. (2022) Rebecca Roelofs, Nicholas Cain, Jonathon Shlens, and Michael C Mozer. 2022. Mitigating bias in calibration error estimation. In _International Conference on Artificial Intelligence and Statistics_, pages 4036-4054. PMLR.
* Schick et al. (2021) Timo Schick, Sahana Udupa, and Hinrich Schutze. 2021. Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in NLP. _Transactions of the Association for Computational Linguistics_, 9:1408-1424.
* Singh et al. (2023) Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Aminesh Garg. 2023. Progrpnt: Generating situated robot task plans using large language models. In _2023 IEEE International Conference on Robotics and Automation (ICRA)_, pages 11523-11530. IEEE.
* Suzgun et al. (2023) Mirac Suzgun, Nathan Scales, Nathanael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Akanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. 2023. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 13003-13051, Toronto, Canada. Association for Computational Linguistics.
* Tam et al. (2024) Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, and Yun-Nung Chen. 2024. Let me speak freely? a study on the impact of format restrictions on large language model performance. In _Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track_, pages 1218-1236, Miami, Florida, US. Association for Computational Linguistics.
* Team et al. (2024) Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Riviere, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. _arXiv preprint arXiv:2403.08295_.
* Thirunavukarasu et al. (2023) Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023. Large language models in medicine. _Nature medicine_, 29(8):1930-1940.
* Tuvron et al. (2023) Hugo Tuvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachatux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLama: Open and efficient foundation language models. _ArXiv_, abs/2302.13971.
* Wang et al. (2024) Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichuan Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. 2024. Matheoder: Seamless code integration in LLMs for enhanced mathematical reasoning. In _The Twelfth International Conference on Learning Representations_.
* Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In _Advances in Neural Information Processing Systems_, volume 35, pages 24824-24837. Curran Associates, Inc.
* Xia et al. (2024) Congying Xia, Chen Xing, Jiangshu Du, Xinyi Yang, Yiihao Feng, Ran Xu, Wenpeng Yin, and Caiming Xiong. 2024. FOFO: A benchmark to evaluate LLMs' format-following capability. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 680-699, Bangokok, Thailand. Association for Computational Linguistics.
* Xie et al. (2022) Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2022. An explanation of in-context learning as implicit bayesian inference. In _International Conference on Learning Representations_.
* Xu et al. (2022) Ying Xu, Dakuo Wang, Mo Yu, Daniel Ritchie, Bingsheng Yao, Tongshuang Wu, Zheng Zhang, Toby Li, Nora Bradford, Branda Sun, Tran Hoang, Yisi Sang, Yufang Hou, Xiaojuan Ma, Diyi Yang, Nasun Peng, Zhou Yu, and Mark Warschauer. 2022. Fantastic questions and where to find them: FairytaleQA 

- an authentic dataset for narrative comprehension. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 447-460, Dublin, Ireland. Association for Computational Linguistics.
* Xu et al. [2024] Ziyang Xu, Keqin Peng, Liang Ding, Dacheng Tao, and Xiliang Lu. 2024. Take care of your prompt bias! investigating and mitigating prompt bias in factual knowledge extraction. _arXiv preprint arXiv:2403.09963_.
* Yang et al. [2024] Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2024. Large language models as optimizers. In _The Twelfth International Conference on Learning Representations_.
* Yang et al. [2018] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2369-2380, Brussels, Belgium. Association for Computational Linguistics.
* Zhou et al. [2023] Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. _arXiv preprint arXiv:2311.07911_.

 



## Appendix A Format-Instruction Following Scorer

```
0: Task \(T\), language model \({\cal M}\), format constraints \(C\), generated output \(Y\).
0: If \(C\) includes wrapping characters, we denote as \(\{W_{1},W_{2}\}\) and \(is\_wrapping=True\).
0:\(output\_type\) is the data type required by \(C\) when \(T\) is not MCQ.
1:if\(is\_wrapping\)then
2:return False if (any of \(\{W_{1},W_{2}\}\not\in Y\)) or (number of \(W_{1}\in Y\) + number of \(W_{2}\in Y\neq 2\)).
3:\(ans\) = Extract string in between \(\{W_{1},W_{2}\}\).
4:else
5:\(ans=Y\)
6:endif
7:if\(T\) is MCQ then
8:if MCQ output type is character identifier then
9: return True if \(ans\in\{A,B,C,D\}\). False otherwise.
10: else
11:return True if \(ans\in\{\text{options' values}\}\). False otherwise.
12:endif
13:else
14:return True if we can parse \(ans\) as an instance of the class \(output\_type\). False otherwise.
15:endif
```

**Algorithm 1** Format-Instruction Following Scorer

Alg. 1 presents our heuristic algorithm for evaluating the format-instruction following capabilities of LLMs, which is used to compute \(F_{C}\) in Eq. (1). The algorithm is divided into two three main parts:

1. [leftmargin=*]
2. **Lines 1-6**. These lines focus on examining the wrapping requirements by verifying the presence and correctness of the specified wrapping tokens.
3. **Lines 7-12**. These lines are dedicated to checking the formats of MCQ answers (SS5.1).
4. **Lines 13-15**. These lines address the remaining formats, including list and mapping formats.

It is worth noting that Alg. 1 is highly adaptable; formats can be added or removed to tailor it for specific downstream applications.

## Appendix B Theoretical Analysis: Reliability of \(EstTrueE\)

### Proof of Thm. 3.1

Proof of Thm. 3.1..: We omit the case when \(FI_{C}=0\) since in that case, we cannot estimate \(TrueE\). By the definition in Thm. 3.1, we have \(S_{C}\) generated answers that satisfy \(C\). Let's denote \(k=S_{C}\) for simplicity. Let's denote \(k\) performance scores of answers satisfying \(C\) as \(x_{1},\cdots,x_{k}\), and \(\bar{x}=\frac{\sum_{i=1}^{k}(x_{i})}{k}\) as the mean. Finally, \(TrueE\) is the population mean of the performance scores, denoted as \(\mu\).

Statement 1: \(EstTrueE\) is consistentFrom Eq. (5), by rewriting \(EstTrueE\), we have \(EstTrueE=\frac{1}{n}.\sum_{i=1}^{k}(x_{i}).\frac{n}{n}=\bar{x}\), which is an unbiased estimator of the average performance \(TrueE\), i.e., \(Bias(\bar{x})=0\) or \(\lim_{k\rightarrow\infty}Bias(EstTrueE)=0\) (1). Now, let's denote the variance of the performance scores as \(\sigma^{2}\), then the variance of \(EstTrueE\) is \(Var(EstTrueE)=Var(\bar{x})=\frac{\sigma^{2}}{n}\) and \(\lim_{k\rightarrow\infty}Var(EstTrueE)=0\) (2). From (1) and (2), by the Sufficient Condition for Consistency (Amemiya, 1985), we conclude that \(EstTrueE\) is a consistent estimator.

Statement 2: \(FI_{C}\) valueLet's denote \(s^{2}=\frac{1}{k-1}\sum_{i=1}^{k}(x_{i}-\bar{x})^{2}\) as the sample variance of the performance scores \(x_{i}\)s. It is well-known that \(\frac{\sqrt{k}(\bar{x}-\mu)}{s}\sim t_{k-1}\). For estimating the population mean \(\mu\) with finite population size \(n\) and the type I error \(\alpha\), we have the margin of error \(\epsilon\)

\[\epsilon \geq t_{\alpha/2,k-1}\cdot\sqrt{\frac{n-k}{n}}\cdot\frac{s^{2}}{k} \tag{9}\]

where \(\frac{n-k}{n}\) is the finite population correction factor. Eq. (9) is equivalent to:

\[k \geq\frac{n-k}{n}\cdot\left(\frac{t_{\alpha/2,k-1}\cdot s}{\epsilon} \right)^{2} \tag{10}\]

which yields

\[k \geq\frac{1}{\frac{1}{n}+\left(\frac{\epsilon}{t_{\alpha/2,k-1}\cdot s} \right)^{2}}. \tag{11}\]

then

\[FI_{C}=\frac{k}{n} \geq\frac{1}{1+n\cdot\left(\frac{\epsilon}{t_{\alpha/2,k-1}\cdot s }\right)^{2}}. \tag{12}\]

**Statement 3: When \(FI_{C}\) approaches 1, \(EstTrueE\) approaches \(TrueE\).** Since \(EstTrueE\) by its definition in Eq. (5) is continuous with respect to \(FI_{C}\) (Eq. (5)), \(S_{C}\) (Eq. (3)) and \(F_{C}\) (Eq. (3)), therefore, we have the equality:

\[\lim_{FI_{C}\to 100}\left(EstTrueE\right)=EstTrueE(FI_{C}=100)= TrueE.\]

\(\Box\)

### Python Codes for Computing Reliability

```
1importnumpyasnp
2fromscipy.statsimportt
3importmath
4
5defcompute_sample_variance(data):
6n=len(data)
7mean=np.mean(data)
8squareddeviations=[(x-mean)**2forxindata]
9sample_variance=sum(squared_deviations)/(n-1)
10returnsample_variance
11
12defis_estimator_reliable(num_FI,list_evalscores,num_samples=20%):
13##timet-statistics####
14alpha=0.05#5%significancelevel
15df=num_FI-1#degreesoffreedom
16alpha_two_tailed=alpha/2
17t_statistic=t.ppf(1-alpha_two_tailed,df)
18
19##ComputeMOLE_FI*******
20epsilon=0.05#5%marginoferror
21s=math.sort(compute_sample_variance(list_eval_scores))
22returnum_FI/num_samples>1/(1+num_samples*(epsilon/(t_statistic*s)))
23##z)
```

Code Listing 1: Python codes for computing the reliability of \(EstTrueE\) with margin of errors 5% performance with a significance level 5%.

 

Detailed Discussions

We give the numerical results and discussions for all figures and points made in the main paper.

### Multiple-choice Question (MCQ) Discussions

We evaluate Gemma, Mistral, and ChatGPT on the MMLU and BBH datasets using two prompting techniques, Zero-shot (ZS) and Zero-shot Chain-of-Thought (ZS-CoT) (SS5.1). The prompts are specified in SSE.1. We report the \(FI_{C},SysE,EstTrueE\) scores. The results are presented in Tab.2. Additionally, Tab.3, Tab.4, and Tab.5 are the distillation results of Tab.2:

1. Tab.3. For each model, we average its \(EstTrueE\) performance overall benchmarks and prompting techniques. For each task, we average the \(EstTrueE\) scores overall models and prompting techniques. The results of this table are plotted in Fig.2 and discussed in SS5.1.
2. Tab.4. The purpose of this table is to compare the FI scores across formats. We average all the FI scores across models and tasks.
3. Tab.5. The purpose of this table is to see whether CoT (Wei et al., 2022) mitigates format bias. We average all the \(EstTrueE\) scores over all models and benchmarks for each ZS and ZS-CoT prompting method.

Format instruction-following bias.The FI scores across formats are illustrated in Tab.4. There is a notable difference between the scores for character-based and textual value-based formats. Among the models, ChatGPT follows the instructions best with FI score \(52.42\%\). Below we present two examples of Gemma and Mistral failing to follow the format instructions:

\begin{table}
\begin{tabular}{l c c} \hline \hline
**MCQ type** & **Char.** & **Text.** \\ \hline  & \multicolumn{2}{c}{**MMLU**} \\ Gamma-7B-it (EstTrue-Acc) & 0.53 / 27.25 & 8.10 / 18.63 \\ Gemma-7B-it (Systematic-Acc) & 0.12 / 10.32 & 0.17 / 4.86 \\ Gemma-7B-it (FI) & 22.47 / 37.87 & 2.10 / 26.09 \\ \hline Mistral-7B-it-v0.2 (EsTrue-Acc) & 46.14 / 49.31 & 8.37 / 8.52 \\ Mistral-7B-it-v0.2 (Systematic-Acc) & 41.59 / 45.94 & 0.17 / 0.19 \\ Mistral-7B-it-v0.2 (Fb) & 90.12 / 93.16 & 2.03 / 2.23 \\ \hline Llama-1.8B-it (EStrue-Acc) & 20.04 / 35.03 & 0.00 / 47.66 \\ Llama-3.1-8B-it (Systematic-Acc) & 1.79 / 27.16 & 0.00 / 29.05 \\ Llama-3.1-8B-it (FI) & 8.93 / 77.54 & 0.00 / 60.95 \\ \hline ChatGPT (EstTrue-Acc) & 68.55 / 45.53 & 54.85 / 59.67 \\ ChatGPT (Systematic-Acc) & 66.20 / 42.22 & 12.71 / 26.31 \\ ChatGPT (FI) & 96.56 / 92.73 & 23.17 / 44.09 \\ \hline \hline  & \multicolumn{2}{c}{**BBH**} \\ Gamma-7B-it (EstTrue-Acc) & 42.11 / 23.05 & 0.00 / 15.11 \\ Gamma-7B-it (Systematic-Acc) & 0.40 / 13.00 & 0.00 / 6.80 \\ Gamma-7B-it (FI) & 0.95 / 56.40 & 0.00 / 45.00 \\ \hline Mistral-7B-it-v0.2 (EsTrue-Acc) & 76.81 / 62.50 & 0.00 / 0.00 \\ Mistral-7B-it-v0.2 (Systematic-Acc) & 21.20 / 22.00 & 0.00 / 0.00 \\ Mistral-7B-it-v0.2 (FI) & 27.60 / 35.20 & 0.00 / 1.60 \\ \hline Llama-3.1-8B-it (EstTrue-Acc) & 0.00 / 63.04 & 0.00 / 48.40 \\ Llama-3.1-8B-it (Systematic-Acc) & 0.00 / 34.80 & 0.00 / 36.40 \\ Llama-3.1-8B-it (FI) & 0.00 / 55.20 & 0.00 / 75.20 \\ \hline ChatGPT (TestTrue-Acc) & 73.03 / 57.14 & 53.63 / 0.00 \\ ChatGPT (Systematic-Acc) & 26.00 / 16.0 & 53.20 / 0.00 \\ ChatGPT (FI) & 35.60 / 28.00 & 99.20 / 0.00 \\ \hline \hline \end{tabular}
\end{table}
Table 2: MCQ results. Red results are unreliable results measured by Thm.3.1 i.e., inequality Eq.6 does not hold.

 

* [150] A. J. van der Laan, M. R. C. and S. G. K. W. (2009) The effect of the change in the response to changes in 

ZS versus ZS-CoT.We investigate the impact of Chain-of-Thought (CoT) (Wei et al., 2022) on mitigating format bias. The average of estimated True Accuracy scores across models and tasks for the ZS prompting and ZS-CoT prompting are shown in Tab. 5. ZS prompting achieves a higher score in the character category (40.90%) compared to ZS-CoT prompting (33.10%). Similarly, for textual value format, ZS prompting scores higher than ZS-CoT prompting. However, the \(BiasF_{o}\) is lower for the ZS-CoT model (\(17.42\%^{2}\)) compared to the ZS model (\(42.42\%^{2}\)), indicating that CoT slightly decreases the format bias.

Reliability of the results.From Tab. 2, we see that \(21/32\) of the estimated \(EstTrue\) results are reliable. The reliability of results in the MCQ output format varies across different models. German-7B-it and Mistral-7B-it show significant unreliability in textual value format, evidenced by numerous red-marked scores due to models not following the format instructions to output correct formats. In contrast, ChatGPT's results are significantly more reliable in the MMLU and BBH benchmarks (\(7/8\)), with only one unreliable result in the BBH textual format output.

### Wrapping Discussions

We examine Gemma, Mistral, and ChatGPT on the MCQ datasets (MMLU,BBH) and generation datasets (GSM8K, HotpotQA, FairytaleQA) utilizing two prompting techniques, Zero-shot (ZS) and Zero-shot Chain-of-Thought (ZS-CoT) (SS5.2). The prompts are also provided in SSE.2. We measure the \(FI_{C},SysE,EstTrueE\). The results are shown in Tab. 6. Furthermore, Tab. 7, Tab. 8 and Tab. 9 are the distillation outcome of Tab. 6:

1. Tab. 7. For each model, we average its \(EstTrueE\) performance overall benchmarks and prompting techniques. For each task, we average the \(EstTrueE\) scores overall models and prompting techniques. This table is plotted in Fig. 3 and discussed in SS5.2.
2. Tab. 8. The purpose of this table is to compare the FI scores across formats. We average all the FI scores across models and tasks.
3. Tab. 9. The purpose of this table is to see whether CoT (Wei et al., 2022) mitigates format bias. We average all the \(EstTrueE\) scores over all models and benchmarks for each ZS and ZS-CoT prompting method.

Format instruction-following bias.The FI scores over formats are provided in Tab. 8. Overall, LLMs exhibit significant format-following bias across formats with a variance of FI scores of \(345.85\%^{2}\). Among the models, ChatGPT follows the instructions best with an average FI Score of \(85.01\%\). The "Special Character" wrapping format has the highest FI score of \(75.05\%\). Following it is the "Placeholder" wrapping format also shows a high FI score of \(72.47\%\), suggesting it is another effective format for ensuring instruction adherence. In contrast, the "Quoting" wrapping format has the lowest FI score of \(18.55\%\). This significant drop compared to other formats suggests that quoting is the least effective method for wrapping instructions, possibly causing confusion or misinterpretation by the models. Below we present two examples of Gemma and Mistral failing to follow the format instructions:

\begin{table}
\begin{tabular}{l c|c|c} \hline \hline  & **Char.** & **Text.** & \(BiasF_{o}\) \\ \hline Zero-shot & 40.90 & 27.88 & 42.42 \\ Zero-shot Chain-of-Thought & 33.10 & 24.75 & **17.42** \\ \hline \hline \end{tabular}
\end{table}
Table 5: MCQ CoT versus non-CoT. This table is the distillation result of averaging all the Zero-shot and Zero-shot Chain-of-Thought scores across models and tasks in Tab. 3.

 



\begin{table}
\begin{tabular}{l c r r r r r r r} \hline \hline


* [150] A. Krizhevsky, S. Bengio, and J. Salakhutdinov. Learning deep neural networks for large-scale image recognition. In _Advances in Neural Information Processing Systems (IJCNN)_, pages 2378-2396, 2014.
* [151] M. Lanzan, R. Fergus, and G. LeCun. Deep residual learning with convolutional neural networks. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1018-1019, 2014.
* [152] D. Kingma and Ba. Adam: a method for stochastic optimization. _arXiv preprint arXiv:1412.6220_ (2014).
* [153] P. Kingma and Ba. Adam: a method for stochastic optimization. _Journal of Machine Learning Research_**49** no. 2, 2015.
* [154] C. Li, X. Zhang, Z. Liu, and Y. Sun. Dropout: a simple network for stochastic optimization. In _NIPS_, volume 1, pages 1098-1109, 2014.
* [155] H. Lin, W. Dong, and T. Darrell. Stochastic gradient descent using recurrent neural networks. In _Neural Computation & Simulation (NIPS'08)_, pages 103-113, 2008.
* [156] I. Goodfellow, I. Sutskever, and J. Pouget. Imagenet classification by jointly connected layers. In _Advances in neural information processing systems_, pages 1038-1040, 2014.
* [157] V. Mnih, E. Rabinovich, and N. Srivastava. Very Large-Scale Convolutional Networks. In _Advances in Neural Information Processing Systems (ICMLSP)_, pages 1-10, 2015.

[MISSING_PAGE_POST]

 

exhibits similar variability, with some unreliable scores in "Quoting" and "Placeholder" formats. ChatGPT generally demonstrates mostly reliable results, with only 1 quoting result unreliable.

### List Discussions

We assess Gemma, Mistral, and ChatGPT with two prompting techniques, Zero-shot (ZS) and Zero-shot Chain-of-Thought (ZS-CoT) (SS5.3) on two benchmarks SciDocsRR and SemEval2017. Our prompts are provided in E.3. We utilize \(FI_{C},SysE,TrueE\) as our evaluation metrics. The results are illustrated in Tab. 10. In addition, Tab. 11, Tab. 12 and Tab. 13 are the distillation results of Tab. 10:

1. Tab. 11. For each model, we average its \(EstTrueE\) performance overall benchmarks and prompting techniques. For each task, we average the \(EstTrueE\) scores overall models and prompting techniques. This table is drawn in Fig. 4 and its discussions are conducted in SS5.3.
2. Tab. 12. The purpose of this table is to compare the FI scores across formats. We average all the FI scores across models and tasks.
3. Tab. 13. The purpose of this table is to see whether CoT (Wei et al., 2022) mitigates format bias. We average all the \(EstTrueE\) scores over all models and benchmarks for each ZS and ZS-CoT prompting method.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & **Spell Character** & **Bolding** & **Italicizing** & **Brackets** & **Parentheses** & **Placeholder** & **Quoting** & \(BasF_{o}\) \\ \hline Zero-shot & 34.31 & 30.66 & 28.28 & 33.82 & 28.77 & 35.93 & 26.09 & **11.39** \\ Zero-shot Chain-of-Thought & 33.25 & 32.65 & 32.19 & 33.09 & 28.36 & 38.37 & 24.46 & 16.22 \\ \hline \end{tabular}
\end{table}
Table 9: Avg.Estimated Accuracy of non CoT versus CoT for wrapping methods. This table is the distillation result of averaging all the Zero-shot and Zero-shot Chain-of-Thought scores across models and tasks in Tab. 6.

 

Format instruction-following bias.From Tab. 12, the "Newline list" format has the highest FI score of \(68.29\%\), while the FI scores of other formats are around \(40-50\%\). Among the models, ChatGPT follows the instructions best with FI score \(83.27\%\). Specifically, ChatGPT excels at following instructions for all formats. In comparison, Mistral performs poorly with "Special character list", while Gemma struggles the most with "Python" and "Bullet point list". Explaining these phenomena, we observe that Gemma often misinterprets the task of converting textual samples into lists of numbers as a coding task, resulting in answers that do not match the desired format or content. We take an example in dataset **SciDocsRR** SSD.1 with detailed prompt as in SSE.3

Given a query, and a list of documents. Query: Declarative Terrain Modeling for Military Training Games Documents: [...] You are required to transform the list of documents into a binary list of 1 or 0 where 1 indicates the document brings useful information to the topic, and 0 indicates the document does not bring useful information to the topic. Generate your binary list as a Python list

```
Givenaquery,andalistofdocuments. Query:DeclarativeTerrainModelingforMilitaryTrainingGames Documents: [...]
```

Listing 11: Avg estimated true F1 (SemEval2017) and MAP (SciDocsRR) scores over benchmarks (SemEval2017, SciDocsRR) and models (Gamma-7B-it, Mistral, ChatGPT) across 4 widely used list formats (Python, Bullet, Special Character, Newline)

ZS versus ZS-CoT.The results, detailed in Tab. 13 indicate that prompting with ZS-CoT substantially enhances model performance across various formats. Moreover, ZS-CoT effectively reduces format bias, as evidenced by the \({BiasF_{o}}\) metric decreasing from about \(54\%^{2}\) to \(22\%^{2}\). From this, we conclude that

\begin{table}
\begin{tabular}{l c c c|c|c} \hline \hline  & **Python** & **Bullet** & **Special Character** & **Nevline** & \({BiasF_{o}}\) **(Var)** \\ \hline \multicolumn{5}{c}{Models} \\ \hline Germana & 17.12 & 18.25 & 15.12 & 16.21 & **1.32** \\ Mistral & 41.98 & 5.80 & 3.39 & 42.37 & 353.80 \\ LLama & 16.73 & 9.76 & 0.00 & 11.49 & 36.64 \\ ChatGPT & 35.76 & 33.73 & 40.50 & 39.03 & 7.08 \\ \hline Average & 27.90 & 16.88 & 14.75 & 27.27 & 46.97 \\ \hline \multicolumn{5}{c}{Tasks} \\ \hline SemEval2017 & 20.54 & 6.09 & 7.82 & 13.49 & **31.86** \\ SciDocsRR & 35.25 & 27.67 & 21.68 & 41.06 & 54.12 \\ \hline Average & 27.90 & 16.88 & 14.75 & 27.27 & 46.97 \\ \hline \hline \end{tabular}
\end{table}
Table 11: Avg estimated true F1 (SemEval2017) and MAP (SciDocsRR) scores over benchmarks (SemEval2017, SciDocsRR) and models (Gemma-7B-it, Mistral, ChatGPT) across 4 widely used list formats (Python, Bullet, Special Character, Newline) 

CoT reduces format bias.

Reliability of the results.From Tab. 10, \(67\%\) of the \(EstTrue\) results are reliable. However, some scores of Gemma-7B-it and LLama on these benchmarks are red-marked, indicating unreliable results of this model. In contrast, the ChatGPT's results are perfectly reliable.

### Mapping Discussions

We select Gemma, Mistral, and ChatGPT for our evaluation, using two prompting techniques: Zero-shot (ZS) and Zero-shot Chain-of-Thought (ZS-CoT) (SS5.4). These models are tested on the SciREX dataset across three difficulty levels: Easy, Medium, and Hard. Detailed prompt specifications are provided in SSE.4. We calculate \(FI_{C},SysE,EstTrueE\) with the results presented in Tab. 14. Furthermore, Tab. 15, Tab. 16 and Tab. 17 are distilled from Tab. 14:

1. Tab. 15. For each model, we average its \(EstTrueE\) performance overall benchmarks and prompting techniques. For each task, we average the \(EstTrueE\) scores overall models and prompting techniques. This table is numerical results of Fig. 5 and discussed in SS5.4.
2. Tab. 16. The purpose of this table is to compare the FI scores across formats. We average all the FI scores across models and tasks.
3. Tab. 17. The purpose of this table is to see whether CoT (Wei et al., 2022) mitigates format bias. We average all the \(EstTrueE\) scores over all models and benchmarks for each ZS and ZS-CoT prompting method.

Format instruction-following bias.From this Tab. 16, noticeably, there is a significant difference in the FI scores of two mapping formats ("JSON" and "YAML"). The average FI scores of the "JSON" mapping format is \(60.98\%\), which is much higher than "YAML" mapping format (\(43.96\%\)). Overall, among models, ChatGPT yields the best result of \(90.40\%\). Mistral fails to follow "YAML" format instruction, e.g. for the question in dataset **SciREX Easy** SSD.1 with detailed promt as in SSE.4

\begin{table}
\begin{tabular}{l|c c c c|c} \hline \hline  & **Python** & **Bullet** & **Special Character** & **Newline** & \(BiasF_{o}\) **(Var)** \\ \hline Zero-shot & 29.19 & 15.13 & 11.05 & 25.45 & 54.48 \\ Zero-shot Chain-of-Thought & 26.61 & 18.64 & 18.46 & 29.09 & **22.40** \\ \hline \hline \end{tabular}
\end{table}
Table 13: Avg estimated true F1 (SemEval2017) and MAP (SciDocskR) scores of non-CoT versus CoT for list formats. This table is the distillation result of averaging all the scores across models and benchmarks in Tab. 10.

 



\begin{table}
\begin{tabular}{l c|c c} \hline \hline
**Mapping type** & **JSON** & **YAML** \\ \hline  & \multicolumn{3}{c}{**SciREX Easy**} \\ Gemma-7B-it (EstTrue-F1) & 14.60 / 20.84 & 18.20 / 0.82 \\ Gemma-7B-it (Systematic) & 3.54 / 3.79 & 3.03 / 0.10 \\ Gemma-7B-it (FI) & 24.24 / 18.18 & 16.64 / 12.12 \\ \hline Mistral-7B-it (EstTrue-F1) & 28.83 / 32.82 & 0.00 / 0.00 \\ Mistral-7B-it (Systematic) & 11.36 / 32.33 & 0.00 / 0.00 \\ Mistral-7B-it (FI) & 39.39 / 98.48 & 0.00 / 3.03 \\ \hline Llama-3.1-8B-it (EstTrue-F1) & 25.85 / 22.86 & 19.07 / 28.57 \\ Llama-3.1-8B-it (Systematic) & 2.35 / 14.55 & 6.36 / 15.15 \\ Llama-3.1-8B-it (Fl) & 9.09 / 63.63 & 33.34 / 53.03 \\ \hline ChatGPT (EstTrue-F1) & 35.99 / 22.40 & 23.63 / 26.60 \\ ChatGPT (Systematic) & 32.72 / 19.69 & 22.92 / 20.15 \\ ChatGPT (FI) & 90.90 / 87.87 & 96.96 / 75.75 \\ \hline \hline  & \multicolumn{3}{c}{**SciREX Medium**} \\ Gemma-7B-it (EstTrue-F1) & 18.17 / 5.27 & 0.00 / 1.87 \\ Gemma-7B-it (Systematic) & 3.03 / 0.88 & 0.00 / 0.17 \\ Gemma-7B-it (Fl) & 16.67 / 16.67 & 18.18 / 9.09 \\ \hline Mistral-7B-it (EstTrue-F1) & 26.48 / 23.81 & 18.97 / 20.83 \\ Mistral-7B-it (Systematic) & 21.27 / 23.81 & 1.15 / 0.25 \\ Mistral-7B-it (FI) & 80.30 / 100.00 & 6.06 / 1.20 \\ \hline Llama-3.1-8B-it (EstTrue-F1) & 40.80 / 35.01 & 31.64 / 27.10 \\ Llama-3.1-8B-it (Systematic) & 9.89 / 28.12 & 25.41 / 21.76 \\ Llama-3.1-8B-it (Fl) & 24.24 / 80.30 & 80.30 / 80.30 \\ \hline ChatGPT (EstTrue-F1) & 29.07 / 27.29 & 36.55 / 22.70 \\ ChatGPT (Systematic) & 28.19 / 26.47 & 21.60 / 22.70 \\ ChatGPT (Fl) & 96.96 / 96.96 & 59.09 / 100.00 \\ \hline \hline  & \multicolumn{3}{c}{**SeiREX Hard**} \\ Gemma-7B-it (EstTrue-F1) & 34.04 / 29.18 & 1.65 / 0.87 \\ Gemma-7B-it (Systematic) & 4.17 / 10.61 & 0.25 / 0.04 \\ Genma-7B-it (Fl) & 12.12 / 36.36 & 15.15 / 4.55 \\ \hline Mistral-7B-it (EstTrue-F1) & 22.44 / 30.34 & 12.54 / 15.95 \\ Mistral-7B-it (Systematic) & 20.40 / 26.66 & 1.71 / 1.58 \\ Distral-7B-it (FI) & 90.90 / 87.87 & 13.63 / 9.90 \\ \hline Llama-3.1-8B-it (EstTrue-F1) & 39.33 / 39.66 & 27.39 / 26.05 \\ Llama-3.1-8B-it (Systematic) & 5.36 / 33.05 & 25.32 / 24.08 \\ Llama-3.1-8B-it (Fl) & 13.63 / 92.42 & 83.33 / 92.42 \\ \hline ChatGPT (EstTrue-F1) & 20.25 / 22.57 & 11.76 / 12.07 \\ ChatGPT (Systematic) & 19.64 / 22.23 & 11.59 / 10.43 \\ ChatGPT (FI) & 96.96 / 98.48 & 98.48 / 86.36 \\ \hline \end{tabular}
\end{table}
Table 14: Mapping results. Red results are unreliable results measured by Thm. 3.1 i.e., inequality Eq. (6) does not hold.

\begin{table}
\begin{tabular}{l l|r r} \hline \hline  & **JSON** & **YAML** & **Average** & \(BiasF_{o}\) **(Var)** \\ \hline  & \multicolumn{3}{c}{**Models**} \\ \hline Gemma & 20.42 & 3.91 & 12.17 & 68.14 \\ Mistral & 27.46 & 11.39 & 19.43 & 64.56 \\ Llama & 33.92 & 26.64 & 30.28 & 13.25 \\ ChatGPT & 26.27 & 22.22 & 24.25 & **4.10** \\ \hline  & \multicolumn{3}{c}{**Tasks**} \\ \hline Easy & 25.53 & 14.61 & 20.07 & 29.79 \\ Medium & 25.74 & 19.96 & 22.85 & **8.36** \\ Hard & 29.77 & 13.54 & 21.66 & 65.87 \\ \hline \hline \end{tabular}
\end{table}
Table 15: Avg estimated true F1 scores over benchmarks (SciREX Easy, SciREX Medium and SciREX Hard) and models (Gamma-7B-it, Mistral, ChatGPT) across 2 widely used mapping formats (JSON and YAML). This table is the distillation result of averaging all the estimated true performance scores across models and benchmarks in Tab. 14.



Extract the entities reflecting the tasks in the following document:

Document: Published as a conference paper at ICLR 2017 QUERY - REDUCTION NETWORKS FOR QUESTION ANSWERING. Your output must bein YAML format: "" Task: [...] """ Wrrap your final answer by <ANSWER> and </ANSWER>. Mistral: "'yaml - TASK_1: Prepare conference paper - TASK_2: Submit conference paper to ICLR 2017 - TASK_3: Research and develop Query-Reduction Networks for Question Answering ", <ANSWER> "'yaml - TASK_1: Prepare conference paper - TASK_2: Submit conference paper to ICLR 2017 - TASK_3: Research and develop Query-Reduction Networks for Question Answering" </ANSWER>.

ZS versus ZS-CoT.From Tab. 17, it is evident that the performance of ZS prompting surpasses that of ZS-CoT for both formats. Upon comparing the \(BiasF_{o}\) across prompting techniques, we conclude that CoT (Wei et al., 2022) does not mitigate format bias. Reliability of the results.From Tab. 14, \(77\%\) of the \(EstTrue\) results are reliable. The reliability of the results in the mapping output format shows variability across different models and formats. Noticeably, "YAML" mapping format results are less reliable than "JSON" ones. On the other hand, ChatGPT illustrates its high reliability in all mapping formats while Mistral-7B-it and Gemma-7B-it are opposite, and all the results in the "YAML" mapping format of these models are unreliable.

### Mitigating Format Bias Results

In this section, we present the numerical results of our proposed techniques for mitigating format biases using ChatGPT on MMLU, as shown in Tab. 18 indexes 1-15.

\begin{table}
\begin{tabular}{c|c c|c c} \hline \hline
**Model** & **JSON** & **YAML** & **Avg.** & \(BiasF_{o}\) \\ \hline  & 20.71 & 12.62 & 16.66 & **16.34** \\  & 82.82 & 5.13 & 43.98 & 1509.00 \\  & Llama & 45.70 & 71.97 & 58.84 & 172.46 \\  & ChatGPT & **94.69** & **86.11** & **90.40** & 18.41 \\ \hline   **Avg.** & 60.98 & 43.96 & 52.47 & 72.45 \\ \hline \hline \end{tabular}
\end{table}
Table 16: Avg FI scores over benchmarks and models across 2 widely used mapping formats (JSON and YAML). This table is the distillation result of averaging all the FI scores across models and benchmarks in Tab. 14.

\begin{table}
\begin{tabular}{l l|c|c} \hline \hline  & **JSON** & **YAML** & \(BiasF_{o}\) \\ \hline Zero-shot & 28.02 & 16.79 & **31.55** \\ Zero-shot Chain-of-Thought & 26.01 & 15.29 & 28.73 \\ \hline \hline \end{tabular}
\end{table}
Table 17: Avg ZS and ZS-CoT scores over benchmarks and models across 2 widely used mapping formats (JSON and YAML). This table is the distillation results across models and benchmarks in Tab. 14.

 

1. **Demonstrations with formats reduce bias (Indexes 7-12).** From Tab. 18 indexes 7-12, we observe that using demonstrations with formats generally increases the average of FI scores, from \(72.99\%\) without any demonstration (index 3), to \(79.20\%\) with using one demonstration and \(85.32\%\) with using \(5\) demonstrations. Moreover, we find that the performance does not scale linearly with the FI score, indicating that simply increasing the FI score does not necessarily improve the models' performance or reduce format biases.
2. **Repeating format instructions reduces format bias (Indexes 4-6).** From Tab. 18 index 6, most of the formats, repeating the format instruction can increase the FI score (compared to index 3), except for the "Placeholder". Manual investigation reveals that repeatedly using the "Placeholder" format confuses the model about the actual location of the placeholder, leading to the model omitting the format. Nevertheless, this strategy generally reduces the format bias by decreasing the variance of results from formats other than "Placeholder", leading to overall reduction.
3. **Fine-tuning with additional format data can eliminate format bias (Indexes 13-15).** Finetuning mostly eliminates the format bias problem of the LLM with the bias score only \(0.71\%^{2}\) from Tab. 18 indexes 13-15, while increasing the average FI score up to almost perfect with \(99.59\%\). This demonstrates that finetuning can help LLMs become more familiar with format tokens and requirements, reducing bias towards different formats. For **Gamma-2B-it** model indexed 16-27, we observe that:

1. **Repeating format instruction (Indexes 19-21).** This approach does not resolve Gamma-2B-it's inability to adhere to the "Blackets" and "Parentheses" instructions. However, disregarding these two formats, the strategy effectively reduces format bias in the model's performance.
2. **Demonstrations (Indexes 22-27).** Demonstrations significantly mitigate format bias, as evidenced by higher FI scores and reduced variance across format performances.



Experimental Details

### Dataset Details

We provide descriptions of all datasets we use in this paper.

MMLU (Hendrycks et al., 2021).MMLU is a benchmark for evaluating the performance of language models on Multiple Choices Question on a wide range of subjects across STEM, the humanities, social sciences, and other areas, testing the model's ability to understand and reason in diverse domains.

BBH (Suzgun et al., 2023).BBH is a MCQ dataset which includes a variety of challenging benchmarks that require advanced reasoning, comprehension, and other complex cognitive skills.

GSM8K (Cobbe et al., 2021).GSM8K is a dataset of 8,000 math word problems designed for grade school students. The problems require not just basic arithmetic but also multi-step reasoning to solve.

HotpotQA (Yang et al., 2018).HotpotQA is a question-answering dataset with a focus on multi-hop reasoning. It contains questions that require finding and combining information from multiple Wikipedia articles to derive the answer.

FairytaleQA (Xu et al., 2022).FairytaleQA is a dataset designed for evaluating narrative comprehension, particularly in the context of children's fairytales. It includes questions that test understanding of characters, plots, and settings in faiyrtales.

SciDocsRR (Cohan et al., 2020).SciDocsRR is a dataset for evaluating information retrieval systems, particularly in the scientific domain. It includes tasks like citation prediction, document classification, and other retrieval-based evaluations.

SemEval2017 (Augenstein et al., 2017).SemEval2017 is part of an ongoing series of evaluations for semantic analysis in natural language processing. It includes a wide range of tasks such as sentiment analysis, semantic textual similarity, and information extraction.

SciREX (Jain et al., 2020).SciREX is a dataset for evaluating models on the task of information extraction from scientific literature. It focuses on extracting entities, relations, and other structured information from research papers.

### Experimental Results

We present the hyperparameters setting for our experiments below.

Gamma-7B-it (Team et al., 2024).For Gemma 7B-it, use the weights from Google and Huggingface2. We use Nucleus Sampling (Holtzman et al., 2020) as our decoding strategy with a \(p\) value of \(0.95\), a temperature value of \(0.1\), and a window size of \(1024\).

Footnote 2: [https://huggingface.co/google/gemma-7b-it](https://huggingface.co/google/gemma-7b-it)

Mistral-7B-it-v0.2 (Jiang et al., 2023).For Mistral 7B-it, use the weights from MistralAI and Huggingface3. We use Nucleus Sampling (Holtzman et al., 2020) as our decoding strategy with a \(p\) value of \(0.9\), and a window size of \(1024\).

Footnote 3: [https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2)

 ChatGPT (gpt3.5-turbo-0125) (OpenAI, 2022).For ChatGPT, we use the system role: "You are helpful assistant!". We set the "max_tokens" to be 1024, "top_p=1", "frequency_penalty=0", "presence_penalty=0", and the model mode is "gtd3.5-turbo-0125".

Datasets for finetuning ChatGPT and finetuning setups.We preprocess the "auxiliary_train" dataset of MMLU (Hendrycks et al., 2021), resulting in the training set of \(6500\) samples as discussed in SS6. We preprocess a small, distinct validation set with the same ratio as the training set among formats "20-20-40-40-50-20-50", resulting in a total of \(240\) samples for validation.

We use the default finetuning setup of OpenAI for ChatGPT. **Our finetuning costs \(63.86\) US$.

 

Prompting

### MCQ Prompt Details

The input for the models is the combination of the following components:

\[{\bf Input}=\textbf{{non-CoT/CoT $\times$ Char./Text.} Instruction}+{\bf Question}+({\bf CoT Wrapping})\]

where **non-CoT/CoT Instruction** shows that model uses Zero-shot or Chain-of-Thought, given that

* **non-CoT \(\times\)****Char. Instruction** = "Answer the following multiple-choice question by outputting only the designated character identifier."
* **non-CoT \(\times\)****Text. Instruction** = "Answer the following multiple-choice question by outputting the textual value of your choice without the character identifier without any textual description."
* **CoT \(\times\)****Char. Instruction** = "Answer the following multiple-choice question step-by-step by outputting only the designated character identifier."
* **CoT \(\times\)****Text. Instruction** = "Answer the following multiple-choice question step-by-step by outputting the textual value of your choice without the character identifier."

**Question** is the main content of the task and **CoT Wrapping** is wrapping instruction if using CoT. i.e. **CoT Wrapping** = "Wrap your final answer by <ANSWER> and </ANSWER>."

### Wrapping Prompt Details

The input for the models is the combination of the following components:

\[{\bf Input}={\bf non-CoT/CoT Instruction}+{\bf Question}+{\bf Wrapping Format Instruction}\]

where **non-CoT/CoT Instruction** shows that model uses Zero-shot or Chain-of-Thought, given that

* If MCQ task (MMLU,BBH) 1. **non-CoT Instruction** = "Answer the following multiple-choice question by outputting only the designated character identifier."
2. **CoT Instruction** = "Answer the following multiple-choice question step-by-step by outputting only the designated character identifier."
* If generation task (GSM8K, HotpotQA, FairytaleQA) 1. **non-CoT Instruction** = "Answer the following question."
3. **CoT Instruction** = "Answer the following question step by step."

**Question** is the main content of the task, and **Wrapping Format Instruction** is the format we want the model to output, detailed as

* **Special Character wrapping** = "Wrap your final answer by <ANSWER> and </ANSWER>."
* **Bolding wrapping** = "Wrap your final answer in bold by enclosing it with double asterisks."
* **Italicizing wrapping** = "Wrap your final answer in italics by enclosing it with single asterisks."
* **Brackets wrapping** = "Wrap your final answer using double square brackets."
* **Parentheses wrapping** = "Wrap your final answer using double parentheses."
* **Placeholder wrapping** = "Wrap your final answer by filling in the placeholder below: 'So the answer is: {{placeholder}}'
* **Quoting wrapping** = "Wrap your final answer using triple quotation marks." 



### List Prompt Details

For dataset **SciDocsRR**, the input for the models is the combination of the following components:

\[\textbf{Input}=\textbf{Information}+\textbf{Requirement}+\textbf{List Format Instruction}\\ ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}





### Mapping Prompt Details

For all three datasets, we use the following formula for the input of the models

\[{\bf Input}={\bf Requirement}+{\bf Document}+{\bf Mapping}~{}{\bf Format}~{}{ \bf Instruction}+({\bf CoT}~{}{\bf Wrapping})\]

where

* **Requirement** = "Extract the entities reflecting the tasks in the following document:" if using non-CoT model and "Extract the entities reflecting the tasks in the following document step-by-step:" if using CoT model
* **Document** is the main content of the task.
* **CoT Warping** = "Wrap your final list by <ANSWER> and </ANSWER>."
* **Mapping Format Instruction** starts with defining a specific format for the model and then instructs the model to follow. In detail, we have 1. For **Easy** dataset, we define: ```
1JSON_FORMAT={  ..Task'^:[...]  2   3   4   5 YAML_FORMAT=""Task:[...]""
2 Then * JSON Mapping = "Your output must be a Python dictionary with the key 'Task' and value as a list of task name entities: {str(JSON_FORMAT)}" * YAML Mapping = "Your output must be in YAML format: {str(YAML_FORMAT)}"
3. For **Medium** dataset, we define:
```  1JSON_FORMAT={  ..Task'^:[...],  .'Method'':[...]  3   5   7 YAML_FORMAT=""Task:[...]  3   8 Method:[...]  4    9

Then * JSON Mapping = "Your output must be a Python dictionary with the keys 'Task' and 'Method', and value is a list of task name entities and method name entities: {str(JSON_FORMAT)}" * YAML Mapping = "Your output must be in YAML format: {str(YAML_FORMAT)}"
3. For **Hard** dataset, we define:

YAML_FORMAT = '......'

Task:[...]

Method:[...]

Material:[...]

Metric:[...]

'....'

Then

* **JSON Mapping** = "Your output must be a Python dictionary with the keys are 'Task', 'Method', 'Material', 'Metric', and value is a list of task name entities, method name entities, material name entities, metric name entities: (str(J50N_FORMAT))"
* **YAML Mapping** = "Your output must be in YAML format: {str(YAML_FORMAT)}