Introduction
To unlock the full potential of automating real-world applications, state-of-the-art large language models (LLMs) (Brown et al., 2020 ; Chowdhery et al., 2022 ; OpenAI, 2022 ; Touvron et al., 2023 ) are increasingly leveraged to tailor outputs to specific task formats. This powerful approach has driven advancements across domains including medicine Thirunavukarasu et al. ( 2023 ); Clusmann et al. ( 2023 ) , data analysis Cheng et al. ( 2023 ); Liu et al. ( 2023 ) , and even evaluating models themselves Chiang and Lee ( 2023 ); Chang et al. ( 2024 ) .
Employing LLMs in such applications heavily depends on not only their format-following capability but also high-quality results within formats .

While many studies, including those listed above, have utilized LLMs to output in specific formats, understanding their format capabilities is critical yet has received limited attention. Recently, Zhou et al. ( 2023 ) and Xia et al. ( 2024 ) introduced benchmarks assessing LLM format-following proficiency. However, these studies neglect deeper insights into how these formats impact model performance, which is the ultimate concern for industrial and practical applications. Given numerous formats recently introduced across tasks and models, assessing this aspect is essential for business yet challenging. Evaluation can be ambiguous and often overlook cases where models provide correct answers but are formatted wrong (Case (1) in Fig. 1 ).

Bridging these gaps, we conduct the first systematic evaluation of the format bias
of LLMs. Our study attempts to answer the research questions:

To fairly assess bias in model performance across formats, it is crucial to evaluate all scenarios depicted in Fig. 1 . Nonetheless, Case (1) is challenging to automatically measure, requiring costly human investigation. Therefore, we propose a reliable estimator for evaluating LLM performance under format constraints without human intervention by considering format-following scores. We start by redefining LLM evaluation metrics into two distinct classes to construct the estimator, as detailed in Section 3.1 .
Accordingly, we define a metric to quantify format bias in LLMs and establish criteria for evaluating methods that successfully mitigate this bias ( Section 3.2 ). Based on these formulations, we present our format evaluation framework, comprising of the widely-utilized categories of multiple-choice question–answer (MCQ; Section 5.1 ), wrapping ( Section 5.2 ), list ( Section 5.3 ) and mapping formats ( Section 5.4 ).

Across 15 widely-used formats, our evaluation with zero-shot and zero-shot chain-of-thought prompting Kojima et al. ( 2022 ) on eight question-answering and reasoning tasks reveals substantial performance and format-instruction following inequalities. To address this, we examine prompting and fine-tuning using synthesized format data techniques that work for both open- and closed-source LLMs. Our study validates that enhancing LLMs’ capabilities to follow format instructions potentially mitigates format bias: (1) Prompting with demonstrations and (2) Repeating format instructions substantially alleviates this bias. Moreover, we investigate (3) Synthesizing limited format data based on our evaluation results for fine-tuning. Our approaches significantly decrease ChatGPT performance variance across wrapping formats from 235.33 to 0.71 ( \%^{2} ) on MMLU Hendrycks et al. ( 2021 ) . Our key contributions are:

We introduce the first systematic framework to evaluate format performance bias in LLMs.

A large-scale evaluation spanning 15 formats, 8 tasks, and 3 models revealing substantial LLM performance variance across formats.

The development of 3 novel prompting and fine-tuning methods to mitigate this bias.