Is Your Large Language Model Knowledgeable or a Choices-Only Cheater? (2024)

Abstract: Recent work shows that large language models (LLMs) can answer multiple-choice questions using only the choices, but does this mean that MCQA leaderboard rankings of LLMs are largely influenced by abilities in choices-only settings? To answer this, we use a contrast set that probes if LLMs over-rely on choices-only shortcuts in MCQA. While previous works build contrast sets via expensive human annotations or model-generated data which can be biased, we employ graph mining to extract contrast sets from existing MCQA datasets. We use our method on UnifiedQA, a group of six commonsense reasoning datasets with high choicesonly accuracy, to build an 820-question contrast set. After validating our contrast set, we test 12 LLMs, finding that these models do not exhibit reliance on choice-only shortcuts when given both the question and choices. Despite the susceptibility of MCQA to high choices-only accuracy, we argue that LLMs are not obtaining high ranks on MCQA leaderboards just due to their ability to exploit choices-only shortcuts.1

heading: 1 Introduction
text: Multiple-choice question answering (MCQA) is a popular task to test the knowledge of large language models (LLMs) (Robinson and Wingate, 2023). However, recent work shows that LLMs surpass majority class baselines in choices-only settings— when no question and just the choices are given in a prompt (Balepur et al., 2024). This raises the question: Do models obtain high ranks in MCQA leaderboards due to their pretraining knowledge or their ability to exploit choices-only shortcuts? Resolving this query is key to ensure that MCQA leaderboards reliably rank the knowledge of LLMs.
To answer this question, we use a variation of contrast sets—small datasets that test if models “pay attention” to perturbed attributes that should
1Our code is available at https://github.com/ nbalepur/mcqa-artifacts
alter the model’s decision (Levesque et al., 2012; Gardner et al., 2020). For our purposes, we need a contrast set containing pairs of MC entries with identical answer choices, but varied questions that lead to distinct answers. For example, in Figure 1 (bottom), the MC entries di and dj have the same choices of “the sun” and “the rain”, but di has a question that answered by “the sun”, and similarly for dj . This design ensures that LLMs relying only on shortcuts or patterns in the choices, while ignoring questions, can perform no better than random chance. If a model ranks highly on an MCQA dataset but largely drops in rank on a contrast set based on this dataset, it would reveal that this model obtains a high rank on the original dataset primarily by employing choices-only shortcuts.
Contrast sets are usually built through manual annotation efforts (Gardner et al., 2020; Srikanth and Rudinger, 2022), as model-generated data can be biased. However, writing MC questions with high-quality distractors is difficult even for experts (Gierl et al., 2017). Further, rewritten questions can exhibit distributional differences from the original questions, altering the difficulty of the MCQA task.
To address this issue, we cast the creation of contrast sets for MCQA datasets to a graph mining task. We treat each MC entry di in the dataset as a vertex in an undirected graph, and draw edges
15
between entries di and dj if the gold answer from di is semantically equivalent to a distractor in dj , and vice versa. For instance, in Figure 1 (top), the gold answer of “rain” in di is semantically similar to the distractor of “rainfall” in dj and vice versa, so we draw an edge between di and dj . Thus, an edge (di, dj) means that the gold answers in di and dj can form a set of choices, with questions in di and dj leading to distinct answers in said choices, mirroring the desired format of our contrast set. We find the maximum matching of this graph to obtain the largest contrast set of distinct MCQA questions derived from the initial dataset. This method overcomes the burden of writing contrast sets, while only minimally using models for semantic equivalence, reducing the risk of model-generated biases.
We use our approach to build an 820-question contrast set from six commonsense MCQA datasets from the UnifiedQA collection (Khashabi et al., 2020). We first ask three annotators to assess our contrast set, finding that it has questions with plausible distractors (§4.1). This finding suggests that we have built a high-quality MCQA contrast set.
After verifying the quality of our contrast set, we test 12 LLMs (Touvron et al., 2023; Penedo et al., 2023; Jiang et al., 2023; Young et al., 2024; Team et al., 2024) on the UnifiedQA evaluation set and its mined contrast set (§4.2). Our LLMs surpass random guessing using just the choices on the original evaluation set, aligning with prior work. (Balepur et al., 2024). However, when prompted with both the question and choices, LLM accuracy rankings between the initial evaluation set and contrast set are highly consistent, with Kendall’s τ near 0.9.
Since no LLM rank drops markedly, we claim that our tested LLMs are not ranking highly on MCQA leaderboards solely due to their ability to exploit choices-only shortcuts. Despite the susceptibility of MCQA to high choices-only accuracy, we argue the task may still reliably rank LLM knowledge. As a result, we recommend that future works continue to explore the behavior of LLMs in choices-only settings to help explain how LLMs can adeptly perform MCQA without the question.

heading: 2 Automatic Contrast Set Creation
text: We assume we are given an MCQA dataset D with data entries di = (qi, Ci, ai), where qi is a question, Ci is a list of choices, and ai ∈ Ci is the gold answer. Our goal is to build a contrast set Dcontr from D to probe if LLMs rely on choice-only shortcuts
in MCQA. Typically, humans manually create contrast sets (Srikanth and Rudinger, 2022; Gardner et al., 2020), as model-generated data can be biased (Yu et al., 2024). However, since writing MCQA problems is challenging even for experts (Offerijns et al., 2020; Gierl et al., 2017), we seek to automatically mine a contrast set Dcontr from the original dataset D without model-generated data.
To automatically build contrast sets, we need MCQA entry pairs in the style of Figure 1—pairs with the same choices C′ = {ai, aj}, but questions qi and qj leading to distinct answers ai and aj in C′, respectively. We define this format as an entry pair pij = ⟨(qi, {ai, aj}, ai), (qj , {ai, aj}, aj)⟩. Creating the largest possible Dcontr with distinct questions is equivalent to finding the maximum set of unique entry pairs pij in D. In the next sections, we outline our graph-based approach to mine entry pairs from the original dataset D to form Dcontr.

heading: 2.1 Graph Representation
text: While a simple strategy to find an entry pair pij is to sample two entries (qi, Ci, ai), (qj , Cj , aj) ∈ D and let C′ = {ai, aj}, this may result in low-quality questions, as there is no constraint that ax and ay form a plausible set of choices (§4.1). For instance, if ai is a ratio and aj is an integer, choices {ai, aj} are implausible and result in a low-quality question. To address this, we intuit that the original dataset D reveals if two answers ai and aj are plausible distractors for each other. For answers ai ∈ Cx and aj ∈ Cy, if ai is semantically equivalent to a distractor c ∈ Cj \ {aj} and likewise for aj and Ci, the set of choices C′ = {ai, aj} will be plausible.
To execute this idea, we represent the dataset D as an undirected graph G. Each entry di ∈ D is a vertex for G. We draw an edge between entries di and dj if the gold answer ai is semantically equivalent to a distractor c ∈ Cj \ {aj} and vice versa, meaning that the choices ai and aj can form a plausible set of choices based on D. We create edges with semantic equivalence over exact match to consider choices with minor differences, like “rain” and “rainfall” in Figure 1, increasing the candidate size of our contrast set. We compute semantic similarity via NLI-based embeddings (Conneau et al., 2017) and set a strict cosine similarity threshold of 0.85 to determine semantic equivalence.

heading: 2.2 Mining Entry Pairs
text: We now mine entry pairs from the graph G to build a contrast set Dcontr. For any edge (di, dj) in G,
we know that ai and aj form a set of plausible choices. When C′ = {ai, aj}, entries (qi, C′, ai) and (qj , C′, aj) form an entry pair pij . Using this idea, we build Dcontr by finding the maximum matching (Boppana and Halldórsson, 1992) of G, which gives the largest set of edges in G where no two edges are adjacent. Each edge in the maximum matching form an entry pair for the contrast set and since no edges are adjacent, each entry pair contains two unique questions. This yields the largest possible contrast set without duplicate questions.

heading: 3 Experimental Setup

heading: 3.1 A Contrast Set for UnifiedQA
text: The purpose of our contrast set Dcontr is to study whether high choices-only accuracy influences the ranking of LLMs on MCQA leaderboards. Hence, Dcontr should be based on a dataset with high accuracy in choices-only settings. The two datasets from Balepur et al. (2024) with the highest choices-only accuracy are commonsense datasets (Clark et al., 2018; Zellers et al., 2019), so we derive Dcontr from an MCQA split of UnifiedQA (Khashabi et al., 2020), which has 7611 questions from six commonsense datasets: ARC (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018), CommonsenseQA (Talmor et al., 2019), QASC (Khot et al., 2020), PIQA (Bisk et al., 2020), and SIQA (Sap et al., 2019). Using our graph mining algorithm (§2), we build an 820-question contrast set. This size aligns with contrast set sizes in prior works, ranging from 600 to 1000 (Srikanth and Rudinger, 2022; Gardner et al., 2020).

heading: 3.2 Prompt Design
text: Few-shot prompting is currently the only method that has shown LLMs can surpass random guessing when given only the choices as input. For our experiments, we follow the few-shot format of Balepur et al. (2024) and use a full prompt (3.1) to assess when LLMs see both the questions and choices, and a choices-only (3.2) prompt for just the choices:
Prompt 3.1: Full Prompt
Question: q Choices:\n(A) ca \n(B) cb \n(C) cc \n(D) cd Answer: a
Prompt 3.2: Choices-Only Prompt
Choices:\n(A) ca \n(B) cb \n(C) cc \n(D) cd Answer: a
In the boxes above, the non-highlighted text represents the model input, while the highlighted text represents the model generation. In the few-shot prompts, exemplars follow the same format shown in the prompt box with the highlighted text replaced by the ground truth (Example in Appendix A.2).

heading: 4 Results

heading: 4.1 Qualitative Analysis
text: To assess the quality of the contrast set produced by our graph mining algorithm, we ask three Ph.D. students in computer science to compare 50 of our questions versus a baseline that randomly picks entry pairs (details in Appendix B.1). These methods only differ by distractors, so following Gierl et al. (2017), we ask annotators to compare the plausibility of the two distractors as a proxy for question quality. All three annotators find that our method has significantly more plausible distractors than the baseline (Figure 2), suggesting that our extracted contrast set from UnifiedQA is high-quality.

heading: 4.2 Are LLMs Knowledgeable or Choices-only Cheaters?
text: Following our quality checks, we use our contrast set to study if high choices-only accuracy influences the ranking of LLMs when questions and choices are given. We assess 6 LLM families on the UnifiedQA evaluation set and our contrast set: LLaMA-2 (Touvron et al., 2023), Falcon (Penedo et al., 2023), Mistral (Jiang et al., 2023), Mixtral (Jiang et al., 2024), Gemma (Team et al., 2024), and Yi (Young et al., 2024). We use 5-shot and 10- shot full and choices-only prompts (Prompts 3.1, 3.2). Appendix A.3 has more prompting details.
On the UnifiedQA evaluation set, our LLMs often surpass random guessing with choices-only prompts (Figure 3, left), aligning with prior work (Balepur et al., 2024). Further, LLMs with higher ranks on the UnifiedQA evaluation set using the full prompt tend to have higher accuracy when using the choices-only prompt, suggesting a correlation between an LLM’s MCQA leaderboard rank and
its ability to exploit choices-only shortcuts. Simply subtracting these values cannot quantify how an LLM performs in MCQA without choices-only shortcuts, since if an LLM can answer a MC question without the question, it does not imply the model is ignoring the question when it has access to the question (Srikanth and Rudinger, 2022).
To better quantify if LLMs are obtaining high ranks on UnifiedQA due to their ability to exploit choices-only shortcuts, we compare model ranks on the original UnifiedQA evaluation set to its contrast set. We note that if a certain LLM relied on choice-only shortcuts substantially more than other models, its contrast set accuracy ranking would largely drop compared to its evaluation set accuracy ranking, as it would be penalized for ignoring the question. However, in the UnifiedQA evaluation set and its contrast set, model rankings of full prompt accuracy are consistent; the 5-shot and 10-shot rankings have Kendall’s τ of 0.88 and 0.91, indicating high consistency. Thus, we claim that the MCQA rankings of our LLMs on UnifiedQA do not primarily stem from their ability to perform well in choices-only settings, and none of our models are considered “choices-only cheaters.”
We find that if an LLM succeeds with choicesonly prompts, it does not imply that this model’s performance in MCQA solely stems from its choices-only abilities. As a result, we believe that despite high choices-only accuracy, MCQA may still be a reliable task to rank the knowledge of LLMs. Further, our results stress the need for more work in explaining how high choices-only accuracy occurs. We believe such efforts are crucial to better interpret LLM knowledge and decision-making.

heading: 5 Related Work
text: Contrast Sets: Contrast sets (Gardner et al., 2020) or counterfactual augmentations (Kaushik et al., 2020; Srikanth et al., 2024), are datasets that probe if models “pay attention” to desired attributes (Elazar et al., 2023). This technique has been applied to many tasks, including natural language inference (Glockner et al., 2018; Ribeiro et al., 2020), story generation (Qin et al., 2019), and ethical judgements (Hendrycks et al., 2021a). While these datasets are often created manually, many works use generation models (Wu et al., 2021; Fryer et al., 2022) to create contrast sets. Instead, we are the first to employ graph mining to build contrast sets, limiting the potential for model-generated biases. MCQA Evaluation: MCQA is a popular testbed not only for benchmarking LLMs (Beeching et al., 2023; Liang et al., 2023), but also for interpreting LLM decision-making. Previous works use MCQA to study prompt sensitivity (Pezeshkpour and Hruschka, 2023; Zheng et al., 2024), logical robustness (Balepur et al., 2023), and recently, the ability to perform MCQA without using the question (Balepur et al., 2024). We give more insights into this last phenomenon by probing if LLMs ignore the question even when it is given in the prompt.

heading: 6 Conclusion
text: We find that while LLMs can perform well in MCQA without access to the question, it does not mean that model rankings on MCQA leaderboards are largely influenced by this ability. This result supports the claim that MCQA can rank the knowledge and ability of LLMs to reason over both questions and choices. Further, we are aligned with recent work that suggests that high choices-only
accuracy does not necessarily imply that models are incapable of true reasoning or comprehension, so we hope future works continue to explore what strategies LLMs may employ to perform well in choices-only settings. Our application of graph mining to MCQA sheds light on one way to do this—the automatic construction of contrast sets— and we hope similar methods can be applied to other tasks to enhance LLM interpretability.

heading: 7 Limitations
text: One limitation lies in the application of our graph mining algorithm solely to the UnifiedQA dataset collection. We choose UnifiedQA for its tendency to elicit high accuracy with choices-only prompts, as commonsense reasoning MCQA datasets have shown to be susceptible to this phenomenon. Since our results show that LLMs rankings are highly consistent on this dataset prone to high choicesonly accuracy, we believe these findings will hold for other MCQA datasets like MMLU (Hendrycks et al., 2021b) with lower choices-only accuracy. However, we invite future research to apply our graph mining algorithm to other datasets, including non-MCQA datasets, to build contrast sets that can further probe LLM decision-making.
Further, we acknowledge that our contrast set contains MCQA questions limited to two choices, diverging from the original evaluation set’s range of two to eight choices. While having less options does make it more likely for a model to guess the right answer, our qualitative analysis shows that the concepts tested in our contrast set are not markedly different in plausibility (§4.1), and thus are not too easy. Further, while our contrast set is easier in theory, it still preserves LLM rankings, even on the subset used to derive the contrast set (Appendix B.3), ultimately supporting the idea that MCQA can reliably rank LLMs capabilities.

heading: 8 Ethical Considerations
text: When models heavily rely on patterns or biases present in datasets, we may overestimate model abilities and face generalizability issues during deployment. In this work, we probe the extent to which LLMs over-rely on patterns in MCQA choices when provided both the question and choices in the prompt, ultimately finding that this effect is small. However, we believe it is still critical for LLM practitioners to be aware that LLMs can outperform random guessing when using just
the choices as input, as this could have downstream effects. We encourage future research efforts in designing special datasets that can help interpret specific abilities within LLM decision-making.
Further, we note that when any model is used in a data creation pipeline, there is the possibility of models propagating their own biases. We specifically address this issue by designing a graph mining algorithm that leverages minimal model intervention, only in the form of computing semantic similarity, which greatly lowers this risk compared to synthetic data generators like LLMs. We hope future works can adopt data creation pipelines with minimal model use similar to ours to avoid the risk of generating model-specific biases or artifacts.

heading: 9 Acknowledgements
text: We would like to thank members of the CLIP lab at the University of Maryland and external collaborators for their feedback and discussions of this work, including Neha Srikanth, Shi Feng, and Jordan Boyd-Graber. We also thank Yu Hou and Dayeon Ki for their help with annotations. This material is based upon work supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE 2236417. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

heading: A Experimental Setup
text: A.1 Dataset Details
The UnifiedQA evaluation set has questions from the evaluation sets of the following six datasets:
• ARC: 1172 four-choice questions drawn from grade-school science questions.
• OpenBookQA: 500 four-choices questions modeled after open-book exams.
• QASC: 926 eight-choice questions about grade school science with a focus on sentence composition.
• CommonsenseQA: 1221 four-choice questions meant to test commonsense knowledge from ConceptNet.
• Physical IQa: 1838 two-choice questions about physical commonsense reasoning.
• Social IQA: 1954 three-choice questions involving reasoning about everyday social interactions.
After running our algorithm, our contrast set contains 377 questions from CommonsenseQA, 285 questions from QASC, 79 questions from ARC, 53 questions from Social IQa, 22 questions from OpenBookQA, and 4 questions from Physical IQa, all of which have two choices.
A.2 Prompt Box Example
The following subsection is adapted directly from the Appendix of Balepur et al. (2024) to highlight the utility of their prompt boxes.
Below, we provide a detailed example to illustrate the application of our prompt boxes. Suppose we have the full prompt (Prompt 3.1):
Prompt 2.1: Full Prompt
Question: q Choices: C Answer: a
In the above prompt, the LLM uses the question q and choices C as input and is asked to generate the letter of the answer a. Suppose we have 5 few-shot examples, with questions q1 ,..., q5, list of choices C1, ..., C5, and ground truth answers a1, ..., a5. The expanded few-shot prompt for the prompt box is written below:
Prompt 2.1: Full Prompt Expanded
Question: q1 Choices: C1 Answer: a1
Question: q2 Choices: C2 Answer: a2
Question: q3 Choices: C3 Answer: a3
Question: q4 Choices: C4 Answer: a4
Question: q5 Choices: C5 Answer: a5
Question: q Choices: C Answer:
Using this prompt, the LLM must generate a, which is the highlighted text in the prompt box.
A.3 Prompting Details
We design few-shot prompts following the format described by our prompt boxes. The few-shot examples were randomly selected from the training set, and we ensured that these contained a balanced distribution of output labels and that the demonstrations were shuffled. We created a few-shot prompt for each dataset. Both the UnifiedQA evaluation set and the contrast set used the exact same prompt. Even though this results in demonstrations with more than two choices, we found that this did not confuse models on the contrast set, as they never outputted an invalid letter (i.e. “(C)” when there are two choices). In the case of an invalid output, which stemmed from a non-letter choice, we marked the output as incorrect.

heading: B Results
text: B.1 Qualitative Analysis Details
Below, we provide the exact instructions (Figure 4) and annotation interface (Figure 5) shown to our annotators. Our annotation interface is based on PrairieLearn (West et al., 2015). Our use of plausibility and relevance for this annotation task is based on existing work (Gierl et al., 2017).
The random baseline we compare against is the trivial solution described in §2.1. This baseline selects a random gold answer from the same dataset to form a set of choices. We apply this algorithm
to the same 50 sampled instances as the ones annotators evaluated with our contrast set, meaning that the questions produced by this baseline only differ by the chosen distractor; the question, choices, and gold answer are all consistent across approaches.
B.2 3-shot Prompting Results In Figure 6, we show the same results as Figure 3 but with three-shot prompting. The same trends of high choices-only accuracy and the consistency of full prompt rankings across evaluation and contrast sets both hold, with a Kendall’s τ of 0.88. We did not test 0-shot prompting as we were working with base LLMs (i.e. unaligned and not instructiontuned), which should not have the capability to complete tasks in a 0-shot manner. We believe that studying choices-only accuracy in 0-shot settings could be an interesting avenue for future work.
B.3 UnifiedQA Evaluation Subset Our mined contrast set only has two choices for every question, while the original evaluation set has questions ranging from 2 to 8 choices. To ensure the consistency of rankings is not confounded by the the reduction of possible choices, we also report the 10-shot accuracy on the subset of UnifiedQA that was used to derive the contrast set. This subset is essentially equivalent to the contrast set, but with additional choices on each question so that the number of choices are consistent. In Figure 7, the UnifiedQA Evaluation set and the UnifiedQA Evaluation subset have a similarly high consistency between rankings of full prompt accuracy. Thus, in our experiments, the number of choices on each question does not seem to largely influence the ranking of LLMs.

heading: Future Work
text: We believe that studying choices-only accuracy in 0-shot settings could be an interesting avenue for future work. B. 3 UnifiedQA Evaluation Subset Our mined contrast set only has two choices for every question, while the original evaluation set has questions ranging from 2 to 8 choices. To ensure the consistency of rankings is not confounded by the the reduction of possible choices, we also report the 10-shot accuracy on the subset of UnifiedQA that was used to derive the contrast set. This subset is essentially equivalent to the contrast set, but with additional choices on each question so that the number of choices are consistent. In Figure 7, the UnifiedQA Evaluation set and the UnifiedQA Evaluation subset have a similarly high consistency between rankings of full prompt accuracy. Thus, in our experiments, the number of choices on each question does not seem to largely influence the ranking of LLMs