RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Related Works
Large language models (LLMs) have shown remarkable proficiency in formatting outputs to meet human expectations. Such formats include markdown for lists and pointers (Achiam et al., 2023 ) , code blocks (Gur et al., 2022 ) , and integrate tags, or LaTeX for scientific texts (Singh et al., 2023 ; Wang et al., 2024 ) . Given the rising importance of formatting capabilities in LLMs, recently, format-following benchmarks have been developed for assessing LLMs’ adherence to specified formats (Zhou et al., 2023 ; Xia et al., 2024 ; Chen et al., 2024 ; Macedo et al., 2024 ; Liu et al., 2024 ) . However, these studies only evaluate format-instruction following capabilities. Our research further assesses LLM performance across different formats, uncovering significant format bias in various tasks and models . We also acknowledge the concurrent work by Tam et al. ( 2024 ) , which examines the impact of format restrictions on LLM performance. However, unlike our approach, they do not disentangle evaluation metrics under format constraints and only evaluate 3 structured formats, substantially fewer than our study.
Theoretical Analysis: Format Evaluation
Automatic evaluation of LLMs in question-answering and reasoning tasks mainly relies on rule-based extraction to identify final answers from generated texts Guo et al. ( 2023 ) .
Within format constraints, determining the model’s true performance, which is our focus, can be ambiguous and inaccurate, as correct responses might be overlooked due to format discrepancies (e.g., Case (1) in Fig. 1 ). To address this, we propose redefining these rule-based evaluation metrics to reliably, transparently and accurately measuring the LLM performance given formats restrictions.
Theoretical Analysis: Format Bias
This section defines the metric to quantify format bias and outlines the criteria to mitigate such bias.
Bias measurement.
To measure the format bias of the LLM \mathcal{M} across k formats F_{o}=\{C_{1},\ldots,C_{k}\} , we define a single metric, BiasF_{o} , as the variance of EstTrueE scores over these k formats, denoted as \{EstTrueE_{1},\ldots,EstTrueE_{k}\} . Let \mu_{EstTrueE}=\frac{1}{k}\sum_{i=1}^{k}EstTrueE_{i} represent the mean EstTrueE score. Then:

BiasF_{o}=\frac{1}{k}\sum_{i=1}^{k}(EstTrueE_{i}-\mu_{EstTrueE})^{2} (8)
Realiability of BiasF_{o} .
By Eq. 8 , the lower BiasF_{o} is, the less format- F_{o} -biased \mathcal{M} is, suggesting a criterion for mitigating output format bias. However, BiasF_{o} is an estimator based on the estimators EstTrueE_{i} . Therefore, to enhance the reliability of BiasF_{o} , it is also necessary to improve the reliability of EstTrueE_{i} by increasing FI_{C_{i}} \forall i ( Theorem 3.1 ). Therefore, we propose two necessary criteria for an effective method to mitigate format bias in LLMs : (i) Minimize bias metric: reducing BiasF_{o} , indicating less format- F_{o} -bias in \mathcal{M} ; (ii) Increase the format-following scores for all formats: ensuring the reliability of BiasF_{o} by increasing the FI scores across all the formats: \{FI_{C_{1}},...,FI_{C_{k}}\} ( Eq. 2 ).
Formats for Evaluation
We establish 4 format categories for evaluation consisting of 15 formats introduced by prior practices:
(i) Multiple-choice question (MCQ) answer ( Section 5.1 ).
where LLMs answer questions by selecting from provided choices, presented as either a (1) Character identifier Robinson and Wingate ( 2023 ) ; or (2) Choice value Chen et al. ( 2023 ) .
(ii) Wrapping ( Section 5.2 ).
where LLMs must enclose the final answer within the two characters, which is crucial for automatic evaluation to isolate the final answer from reasoning thoughts. We focus on evaluating 7 widely used wrapping strategies: (1) Special character Gur et al. ( 2022 ) ; (2) Bolding (Zhou et al., 2023 ) ; (3) Italicizing (Zhou et al., 2023 ) ; (4) Double brackets (Luo et al., 2024 ) ; (5) Double parentheses ; (6) Placeholder (Wang et al., 2024 ) ; (7) Quoting (Zhou et al., 2023 ) .
(iii) List ( Section 5.3 ).
where the output of LLMs is a list of elements. We investigate 4 formats representing lists: (1) Python list Do et al. ( 2023 ) ; (2) Bullet-point list Liu et al. ( 2024 ) ; (3) List of elements separated by a special character ”[SEP]” Boucher ( 2023 ) ; and (4) List of elements arranged on separate lines Mishra ( 2023 ) .
(iv) Mapping ( Section 5.4 ).
where LLMs are employed to output dictionaries or maps. We focus on two ubiquitously used mapping structures: (1) Python dictionary/JSON (JavaScript Object Notation) (Baumann et al., 2024 ) and (2) YAML (Yet Another Markup Language) (Goel et al., 2023 ) .
Format-instruction following.
We introduce Appx.- Alg. 1 , a rule-based heuristic to determine the format-instruction following function F_{C} ( Eq. 1 ) for our benchmarked formats. It calculates the binary FI score by verifying that the generated output includes the specified formatting tokens and that the extracted final answer matches the expected type. It is highly extendable to other formats ( Appendix A ).
Benchmarks.
For MCQ bias evaluation ( Section 5.1 ), we select two datasets: MMLU (Hendrycks et al., 2021 ) and BBH (Suzgun et al., 2023 ) . For MMLU, we randomly choose 27 subcategories. For BBH, we select the sports_understanding category following Gupta et al. ( 2024 ) . For wrapping bias assessment ( Section 5.2 ), in addition to MCQ benchmarks, the following datasets are experimented: GSM8K (Cobbe et al., 2021 ) for reasoning, FairytaleQA (Xu et al., 2022a ) for narrative comprehension, and HotpotQA (Yang et al., 2018a ) for multi-hop reasoning. For list bias investigation ( Section 5.3 ), we use SciDocsRR (Muennighoff et al., 2023 ) , a scientific document ranking task as the order list generation task, and SemEval 2017 (Augenstein et al., 2017a ) , the keyphrase extraction task as the unordered list generation. For mapping bias examination ( Section 5.4 ), we utilize a document-level information extraction task named SciREX (Jain et al., 2020a ) by synthesizing three extraction difficulty levels: easy (extracting from 1 sentence for 1 category), medium ( 3 sentences, 2 categories), and hard ( 5 sentences, 4 categories). For all benchmarks except MCQ, we sample 200 points for evaluation Bai et al. ( 2023 ) .
Models.
We select both open- and closed-source LLMs for our evaluation: Gemma-7B-it (Team et al., 2024 ) and Mistral-7B-it-v0.2 (Jiang et al., 2023 ) for open-source as they are among state-of-the-art open-source LLMs; ChatGPT (gpt-3.5-turbo-0125) for closed-source as this premier chatbot possesses superior instruction-following ability. Our purpose is not to reproduce the models’ performance, but to show the bias.
Metrics.
Following our discussion in Section 3.1 , we disentangle Accuracy (Acc) for MMLU and BBH (Guo et al., 2023 ) ; F1 for GSM8K, HotpotQA, FairytaleQA; and Mean Average Precision (MAP) for SciDocsRR (Muennighoff et al., 2023 ) and we report the metrics EstTrueAcc , EstTrueF1 , EstTrueMAP ( Eq. 5 ) in the main text. For metrics’ reliability, we set \alpha = \epsilon=5\% .
Prompting baselines.
Our focus is on two widely used prompting baselines: (1) Zero-shot (ZS) prompting and (2) Zero-shot Chain-of-Thought (ZS-CoT) prompting (Kojima et al., 2022 ) . For the ZS baseline, we instruct LLMs to answer the question with the prompt “Answer the following question…” followed by the suffix “without any explanation”.
For ZS-CoT, we use the suffix “step-by-step” instead. For the ZS-CoT experiments in Sections 5.1 , 5.3 and 5.4 , LLMs are instructed to wrap the final answer by “<ANSWER>” and “</ANSWER>” tokens to distinctly isolate it from the reasoning chains (see Tab. 1 for the wrapping instruction). We use this wrapping method since our experiment in Section 5.2 shows that it achieves the highest instruction-following score on average across LLMs. Detailed prompts are provided in Appendix E . We average the performance under two prompting methods to report in the main text.
Format Evaluation Experiments
Overall, we find that: (1) Models show substantial format-following bias across formats for all benchmarks; (2) For all models and datasets, significant performance bias exists across formats; (3) 77.67\% of the EstTrue results are reliable, with 16/24 for MCQ, 169/210 for wrapping, 35/48 for list, and 27/36 for mapping formats. We dive into (2) for every format as it is our main focus, (1, 3) are discussed in detail in Sections C.1 , C.2 , C.3 and C.4 .
Setups.
We investigate the bias of LLMs towards different MCQ output formats. We assess two formats as introduced in Section 3.3 : (1) Character identifier and (2) Choice value. For example, if the choice is “[A. Yes, B. No]”, then the character identifier can be “A/B”, while the choice value can be “Yes/No”. We exclude the format combining the character identifier and choice value (such as “A. Yes”) from our evaluation because instructing LLMs to output this format can be non-trivial and require manual effort to craft instructions tailored for different models. To ensure that LLMs understand the “Character identifier” and “Choice value” as we expect, we add a contrastive format requirement to the prompts (e.g., “without any textual description” for the “Character identifier” prompts).
Results.
Fig. 2 provides a synopsis of our evaluation results, with numerical values shown in Appx.- Tab. 2 . From Fig. 2 -left, we observe that Mistral possesses the highest disparity between the two MCQ answer formats, with 58.69\% accuracy on average for character and only 4.22\% for textual value. Additionally, despite ChatGPT often being regarded as one of the most robust LLMs, it shows a significant performance difference between the two formats ( 19.03\% ). Overall, LLMs are heavily biased towards outputting character identifiers. Requiring them to generate the choice’s value causes notable performance drops of 28.76\% on average.

From Fig. 2 -right, we notice that the models exhibit higher bias on BBH, which appears to be an easier benchmark than MMLU. We attribute this to the small size of BBH, which makes the performance more sensitive to format variations.
Why such bias?
We hypothesize the root cause of the significant performance bias across different formats is the format token bias of LLMs. The non-uniform distribution of FI scores among formats suggests that the models assign probabilities to format instructions differently based on their training data. This leads to varying prior assignments of probabilities to specific tokens, causing final predictions non-uniformly distributed across formats. This hypothesis is supported by our simple fine-tuning with formatted data, which familiarizes LLMs with format instructions relatively equally leading to a drastic format bias reduction ( Section 6 ). This emphasizes the necessity of more research in fine-tuning LLMs to reduce format bias and raises concerns about the reliability and reproducibility of recent studies using varied formats.
Setups.
We study LLM bias towards 7 wrapping methods: (1) Special character; (2) Bolding; (3) Italicizing; (4) Brackets; (5) Parentheses; (6) Placeholder; (7) Quoting, detailed in Tab. 1 . We evaluate LLM performance across formats on the MMLU, BBH, GSM8K, FairytaleQA, and HotpotQA.
Results.
Fig. 3 outlines an overview of our evaluation outcomes with results in Appx.- Tab. 6 .
From Fig. 3 -left, we see that Gemma exhibits the highest bias towards different formats with a BiasF_{o} value ( Eq. 8 (variance) of 56.33\%^{2} , while ChatGPT performs the best with only 12.26\%^{2} . Notably, for “Quoting” and “Parenthesis”, the Gemma follows instructions only about 0-4\% yielding nearly zero performance, highlighting its critical weaknesses. Among the 7 formats, “Placeholder” ( 35.92\% ) proves to be the most effective wrapping output format, while “Quoting” ( 23.74\% ), “Parenthesis” ( 28.64\% ) are among those that achieve the lowest performance.

From Fig. 3 -right, models exhibit bias across all tasks, with the lowest on GSM8K ( 12.97\%^{2} ) possibly because the models were trained on (part of) it, and the highest on BBH ( 70.16\%^{2} ), the challenging task without train data. This demonstrates the pervasive presence of wrapping bias in LLMs.
Why such bias?
The format token bias of LLMs as explained in Section 5.1 is also our hypothesis. Specifically, we found the low performance of the “Quoting” and ”Parenthesis” because, in generation tasks, models often wrap (via quoting/parenthesizing) not only the final answer, as instructed, but also parts of the context (e.g., “‘The answer is 3.”’), leading to poor F1 scores. Moreover, Gemma completely ignores the above format instructions, resulting in 0% FI scores, which also contribute to the low average estimated F1 scores. These strongly indicate the presence of format token bias.
Setups.
We explore the bias of LLMs in generating lists following 4 formats: (1) Python list, (2) Bullet-point list, (3) Character-separated list, and (4) Newline-separated list. We evaluate the models on two list generation tasks: (i) Unordered list , using the keyphrase extraction task on the SemEval 2017 dataset, and (ii) Ordered list , using the document ranking problem on the SciDocsRR task.
Results.
Fig. 4 displays the key findings of our evaluation across models and datasets with numerical results in Appx.- Tab. 10 . From Fig. 4 -left, we notice that Mistral exhibits the most bias, with the BiasF_{o} value ( Eq. 8 ) of 353.80\%^{2} . In contrast, ChatGPT and Gemma show much lower bias, with values of 7.08\%^{2} and 1.32\%^{2} , respectively. Of the four formats, the “Python” and “Newline-separated” formats yield the highest performance, likely due to models trained extensively on code data. Conversely, the “Bullet-point list” format results in the lowest performance, particularly for Mistral, highlighting the inherent bias for such formats.

The performance bias is regardless of the task as plotted in Fig. 4 -right, with the highest BiasF_{o} value of 67.07\%^{2} on the order list generation task SciDocsRR, and significantly lower ( 27.58\%^{2} ) on SemEval2017 task. The high bias in the SciDocsRR task is because Mistral and Gemma mostly failed to perform this task following the “Bullet” and “Special character” list formats while excelling in solving it following the other formats.
Why such bias?
We attribute the bias to the format token bias ( Section 5.1 ). Since the models were extensively trained on code data, they excel in solving code-related instructions. In contrast, “Bullet-point” and “Special character” lists are much less common. One interesting case is Gemma where it performed worse on generating “Python” lists compared to “Bullet-point” lists. Our analysis suggests that Gemma misinterprets the format instruction as a coding request, generating Python code programs instead of an answer in a Python list, suggesting Gemma was predominantly trained on code data.
Setups.
We examine the performance bias of LLMs on two mapping formats as discussed in Section 3 : (1) Python dictionary/JSON; (2) YAML. We preprocess the SciREX task Jain et al. ( 2020a ) as described in Section 4 into three extraction levels: (i) Easy (1 sentence, “Task” category); (2) Medium (3 sentences, “Task, Method”); (3) Hard (5 sentences, “Task, Method, Material, Metric” categories).
Results.
Fig. 5 illustrates a summary of our evaluation with numerical details in Appdx.- Tab. 14 . From Fig. 5 -left, Gemma is the most biased, with a performance gap of 16.51\% between the two formats, followed by Mistral with a 16.07\% gap. ChatGPT, however, is relatively robust against format variations, exhibiting a gap of only 4.05\% . On average, JSON performs significantly better than YAML for mapping, likely because more JSON data is used to train models due to its popularity.

From Fig. 5 -right, extracting 4 categories in the Hard task shows the largest performance gap between mapping formats. Surprisingly, the Medium task displays the least bias, likely because models perform best in this task.
Why such bias?
The bias is attributed to the format token bias ( Section 5.1 ). While Mistral excels in generating JSON, it and Gemma struggle with YAML. Even successfully generating YAML output, Mistral and Gemma frequently introduce noisy information (88%-65% for Mistral with and without CoT, 98%-79% for Gemma) in the response (e.g., a key “Task" should have multiple values, Mistral generates multiple key-value pairs instead e.g., “Task_1:Training \cdots Task_2: \cdots ”), resulting in poor overall performance.
Mitigating Performance Format Bias: Actionable Recommendations
We propose methods as actionable recommendations for mitigating format bias. Generally, three primary streams of techniques have been widely studied and applied to tackle LM biases: (1) Prompting Xu et al. ( 2024 ); Macedo et al. ( 2024 ) ; (2) Calibrating Roelofs et al. ( 2022 ); Li et al. ( 2024 ) ; and (3) Fine-tuning Schick et al. ( 2021 ); Ghaddar et al. ( 2021 ) . While calibration techniques can only be used for white-box models, prompting and fine-tuning can be applied for both black-box (via API) and white-box ones. Therefore, we explore prompting and fine-tuning techniques to reduce format bias. We target mitigating the format bias of ChatGPT , the strongest model that we benchmarked, on MMLU . We aim to reduce the wrapping bias ( Section 5.2 ) due to resource limits, but our methods can be generalized to any model and format.
Demonstration(s) reduce(s) format bias.
As discussed in Section 5.1 , LLMs show bias across formats possibly because of the token bias issue, causing LLMs to non-uniformly comprehend the format instructions. To address this, we examine whether demonstrations with formats can reduce such bias, as they are commonly utilized to enhance LLM’s comprehension of the task patterns Xie et al. ( 2022 ) . Particularly, for each wrapping format in Section 5.2 , we select 1 and 5 random samples from the auxiliary train data of MMLU and manually format the answers as demonstrations. The results are outlined in Fig. 6 . Firstly, incorporating demonstrations typically enhances the FI scores (i) (from 72.99% to 79.19% and 85.31%) of the model, with five demonstrations yielding the most. Secondly, we observe a notable decrease in the BiasF_{o} score (ii) upon supplementing demonstrations. From (i), (ii) and Section 3.2 , we conclude integrating demonstrations mitigates format bias.
Repeating format instructions reduces format bias.
We found that repeating instructions generally increases FI scores (i) across most formats except “Placeholder”, which can consequently lessen the mode’s token bias towards format instructions ( Section 5.1 ). Using our two proposed criteria for effective format bias mitigation in Section 3.2 , it is worth examining if this approach reduces BiasF_{o} , thereby being an effective mitigation. Our answer is yes. By repeating the wrapping instructions of ChatGPT thrice, we observed a decrease in the BiasF_{o} (ii) score presented in Fig. 6 . Combining (i) and (ii) suggests that this strategy is an effective mitigation. For "Placeholder," human investigation reveals that multiple placeholder instructions cause ChatGPT to be confused about where the placeholder is, making it frequently misunderstand and fail to follow this format instruction.
Fine-tuning with additional format data can eliminate format bias.
We hypothesize that completely solving the format token bias problem of LLMs necessitates finetuning them on format data so that they are familiar with tokens in format instructions evenly. We propose a simple data synthesis strategy for finetuning LLMs: we sample a small set of training data for all evaluated formats, with ratios inversely proportional to their systematic evaluation scores ( Section 3.1 ). We chose SysE scores over the EstTrueE because they reflect the current model performance. Practically, based on ChatGPT’s zero-shot systematic performance on MMLU colored in blue in Appx.- Tab. 6 , we approximate the formats’ performance ratios as “ 1,1,\frac{1}{2},\frac{1}{2},\frac{1}{3},1,\frac{1}{3} ” from left-to-right, resulting in training data ratios of formats of “ 1,1,2,2,3,1,3 ”. We then preprocess the MMLU auxiliary training data according to these ratios, scaled by 500 ( 6500 samples total), and train ChatGPT on this dataset.
The finetuned results are plotted in Fig. 6 . Firstly, after finetuning, the average FI score across all formats is nearly perfect at 99.58\% (ii). Secondly, the BiasF_{o} score is significantly reduced from 235.33\%^{2} to 0.71\%^{2} (ii). These (i) and (ii) indicate finetuning largely eliminates format bias.
Conclusions
We introduce the pioneering systematic investigation of format bias in LLM performance, revealing significant biases across widely used formats for all models and benchmarks. Our method involves developing metrics to assess this bias and establishing criteria for effective mitigation. We then introduce prompting and fine-tuning techniques to alleviate format bias based on our evaluation findings. Our work aims to sharpen the focus of future LLM research toward fairer and more robust development.
Limitations
Our study has several limitations. Firstly, the metrics EstTrue and BiasF_{o} proposed in Section 3.1 and Section 3.2 are estimators, not exact measures. As discussed, determining TrueE ( Eq. 4 ) is infeasible, especially for large-scale experiments across various models and datasets. Achieving this would require extensive fine-tuning and comprehensive human evaluations, both prohibitively expensive and impractical in many scenarios. Our proposed metrics EstTrue and BiasF_{o} are handy for large-scale experiments with multiple models and datasets due to their fully automatic nature. We further propose Theorem 3.1 to validate the reliability of TrueE statistically.

Secondly, our empirical evaluation of format bias is limited by computational and budget constraints to specific datasets, formats, and models. This restriction limits the generalizability of our findings and may obscure further insights that could be gained from expanding the experiments to include more formats, larger-scale datasets, and additional task categories.

Finally, while our study primarily attributes format bias to token bias in the training data of LLMs and proposes data-focused approach, it does not extensively explore other factors related to model architecture and training processes. This omission represents a significant area for future research, as more fundamental, architecture-level solutions could be crucial, for addressing format bias in LLMs. Our study underscores the importance of continued research dedicated to quantifying and mitigating format bias.
Ethical Considerations
Our work uncovers significant format bias in LLMs, raising concerns regarding fairness and potential discrimination in real-world applications.
Bias and fairness.
Format bias in LLMs can result in unfair treatment, especially in tasks where multiple possible formats can be used. Our research suggests ways to identify and mitigate format bias, aiming for fairer and more equitable LLM applications.
Societal impact.
Format bias in LLMs has the potential to disproportionately impact specific populations, as different demographics may have preferences for different communication formats. Further research is essential to fully understand its societal implications and ensure fairness across diverse demographics.
Acknowledgements
This research is partially supported by the National Research Foundation Singapore under the AI Singapore Programme (AISG Award No: AISG2-TC2023-010-SGIL) and the Singapore Ministry of Education Academic Research Fund Tier 1 (Award No: T1 251RES2207). DXL and TS are supported by the A*STAR Computing and Information Science (ACIS) scholarship. We thank Goh Yisheng for his contribution in the initial stage of the project. We thank members of WING and Deep Learning Lab, NUS, as well as the anonymous reviewers for the constructive feedback.