RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
E.4 Mapping Prompt Details
For all three datasets, we use the following formula for the input of the models

\textbf{Input}= \textbf{ Requirement}+\textbf{Document}+\textbf{Mapping Format Instruction}+(\textbf{CoT Wrapping})

where

Requirement = “Extract the entities reflecting the tasks in the following document:” if using non-CoT model and “Extract the entities reflecting the tasks in the following document step-by-step:” if using CoT model

Document is the main content of the task.

CoT Wrapping = “Wrap your final list by <ANSWER> and </ANSWER> .”

Mapping Format Instruction starts with defining a specific format for the model and then instructs the model to follow. In detail, we have

For Easy dataset, we define:

Then

JSON Mapping = “Your output must be a Python dictionary with the key ‘Task’ and value as a list of task name entities: {str(JSON_FORMAT)} ”

YAML Mapping = “Your output must be in YAML format: {str(YAML_FORMAT)} ”

For Medium dataset, we define:

Then

JSON Mapping = “Your output must be a Python dictionary with the keys ‘Task’ and ‘Method’, and value is a list of task name entities and method name entities: {str(JSON_FORMAT)} ”

YAML Mapping = “Your output must be in YAML format: {str(YAML_FORMAT)} ”

For Hard dataset, we define:

Then

JSON Mapping = “Your output must be a Python dictionary with the keys are ‘Task’, ‘Method’, ‘Material’, ‘Metric’, and value is a list of task name entities, method name entities, material name entities, metric name entities: {str(JSON_FORMAT)} ”

YAML Mapping = “Your output must be in YAML format: {str(YAML_FORMAT)} ”
Figure 1: A MMLU example Hendrycks et al. ( 2021 ) with ChatGPT across different formats. In Case (1), the model can answer the question but fails to bold only the answer, hindering automatic evaluation. In Case (2), the model follows the format but produces an incorrect result. In Case (3), the model yields the correct answer and format. These show bias in ChatGPT’s performance across formats.; A MMLU example Hendrycks et al. ( 2021 ) with ChatGPT across different formats. In Case (1), the model can answer the question but fails to bold only the answer, hindering automatic evaluation. In Case (2), the model follows the format but produces an incorrect result. In Case (3), the model yields the correct answer and format. These show bias in ChatGPT’s performance across formats.
Figure 2: Average estimated true accuracy ( Section 3.1 ) results of MCQ benchmarks across models (left) and datasets (right) showing performance bias of LLMs across formats.; Average estimated true accuracy ( Section 3.1 ) results of MCQ benchmarks across models (left) and datasets (right) showing performance bias of LLMs across formats.
(a); (b); Average estimated true Accuracy (MCQ) and F1 (GSM8K, HotpotQA, FairytaleQA) scores ( Section 3.1 ) across models (left) and across benchmarks (right), showing performance bias of LLMs across 7 widely used wrapping methods.
(a); (b); Average EstTrueF1 (SemEval2017) and EstTrueMAP (SciDocsRR) ( Section 3.1 ) across models (left) and benchmarks (right) showing performance difference of LLMs across 4 widely used list formats.
Figure 5: Average estimated true F1 scores ( Section 3.1 ) across models (left) and benchmarks (right) showing performance bias of LLMs across 2 widely used mapping formats.; Average estimated true F1 scores ( Section 3.1 ) across models (left) and benchmarks (right) showing performance bias of LLMs across 2 widely used mapping formats.
Figure 6: More demonstrations and repeating format instructions mitigate format bias. Finetuning mostly eliminates the format bias. The performance is reported using ChatGPT on MMLU (Appx.- Tab. 18 for num. results).; More demonstrations and repeating format instructions mitigate format bias. Finetuning mostly eliminates the format bias. The performance is reported using ChatGPT on MMLU (Appx.- Tab. 18 for num. results).
Wrapping “start” and “end” tokens with instructions.
MCQ output format results. Red results are unreliable results measured by Theorem 3.1 i.e., inequality Eq. 6 does not hold.
Average. estimated true accuracy results of MCQ
benchmarks across models (Gemma-7B-it, Mistral, ChatGPT) and datasets (MMLU, BBH) with two MCQ formats (Character and Textual Value). This table is the distillation result of averaging all the estimated true performance scores across models ( Models ) and tasks ( Tasks ) in Tab. 3 .
MCQ format-instruction following scores across formats. This table is the distillation result of averaging all the FI scores across tasks and prompting techniques in Tab. 3 .
MCQ CoT versus non-CoT. This table is the distillation result of averaging all the Zero-shot and Zero-shot Chain-of-Thought scores across models and tasks in Tab. 3 .
Wrapping output format results. Red results are unreliable results measured by Theorem 3.1 i.e., inequality Eq. 6 does not hold.
Avg. estimated true accuracy results of
benchmarks across models (Gemma-7B-it, Mistral, ChatGPT) and datasets (MMLU, BBH) with seven Wrapping format (Special Character, Bolding, Italicizing, Brackets, Parentheses, Placeholder, Quoting). This table is the distillation result of averaging all the estimated true performance scores across models and benchmarks in Tab. 6 .
Avg. Following Instruction Score over all the wrapping formats. This table is the distillation result of averaging all the FI scores across models and benchmarks in Tab. 6 .
Avg.Estimated Accuracy of non CoT versus CoT for wrapping methods. This table is the distillation result of averaging all the Zero-shot and Zero-shot Chain-of-Thought scores across models and tasks in Tab. 6 .
List output format results. Red results are unreliable results measured by Theorem 3.1 i.e., inequality Eq. 6 does not hold.
Avg estimated true F1 (SemEval2017) and MAP
(SciDocsRR) scores over benchmarks (SemEval2017, SciDocsRR) and models (Gemma-7B-it, Mistral, ChatGPT) across 4 widely used list formats (Python, Bullet, Special Character, Newline)
Avg Following Instruction scores over benchmarks (SemEval2017, SciDocsRR) and models (Gemma-7B-it, Mistral, ChatGPT) across 4 widely used list formats (Python, Bullet, Special Character, Newline). This table is the distillation result of averaging all the
FI scores across models and benchmarks in Tab. 10 .
Avg estimated true F1 (SemEval2017) and MAP
(SciDocsRR) scores of non-CoT versus CoT for list formats. This table is the distillation result of averaging all the scores across models and benchmarks in Tab. 10 .
Mapping output format results. Red results are unreliable results measured by Theorem 3.1 i.e., inequality Eq. 6 does not hold.
Avg estimated true F1 scores over benchmarks
(SciREX Easy, SciREX Medium and SciREX Hard) and models (Gemma-7B-it, Mistral, ChatGPT) across 2 widely used mapping formats (JSON and YAML). This table is the distillation result of averaging all the estimated true performance scores across models and
benchmarks in Tab. 14 .
Avg FI scores over benchmarks and models across 2 widely used mapping formats (JSON and YAML). This table is the distillation result of averaging all the FI scores across models and benchmarks in Tab. 14 .
Avg ZS and ZS-CoT scores over benchmarks and models across 2 widely used mapping formats (JSON and YAML). This table is the distillation results across models and benchmarks in Tab. 14 .
Supplementing demonstrations, repeating format instructions, and extra fine-tuning with formats’ data reduce format bias. Performance of ChatGPT on MMLU. All results are reliably measured by Theorem 3.1 i.e., inequality Eq. 6 holds.