ComparisonQA: Evaluating Factuality Robustness of LLMs Through Knowledge Frequency Control and Uncertainty
Authors: Qing Zong, Zhaowei Wang, Tianshi Zheng, Xiyu REN, Yangqiu Song
Accepted at: acl25

COMPARISONQA: Evaluating Factuality Robustness of LLMs Through Knowledge Frequency Control and Uncertainty

Qing Zong, Zhaowei Wang, Tianshi Zheng,
Xiyu Ren, Yangqiu Song
Department of Computer Science and Engineering, HKUST
{qzong, yqsong}@cse.ust.hk

Abstract

The rapid development of LLMs has sparked extensive research into their factual knowledge. Current works find that LLMs fall short on questions around low-frequency entities. However, such proofs are unreliable since the questions can differ not only in entity frequency but also in difficulty themselves. So we introduce COMPARISONQA benchmark, containing 283K abstract questions, each instantiated by a pair of high-frequency and low-frequency entities. It ensures a controllable comparison to study the role of knowledge frequency in the performance of LLMs. Because the difference between such a pair is only the entity with different frequencies. In addition, we use both correctness and uncertainty to develop a two-round method to evaluate LLMs’ knowledge robustness. It aims to avoid possible semantic shortcuts which is a serious problem of current QA study. Experiments reveal that LLMs, including GPT-4o, exhibit particularly low robustness regarding low-frequency knowledge. Besides, we find that uncertainty can be used to effectively identify high-quality and shortcut-free questions while maintaining the data size. Based on this, we propose an automatic method to select such questions to form a subset called COMPARISONQA-Hard, containing only hard low-frequency questions.1

1 Introduction

The rapid advancement of large language models (LLMs) has promoted a lot of study on their factual knowledge and reasoning ability (Wei et al., 2024a,b; Hendrycks et al., 2021).

Sun et al. (2024) and Mallen et al. (2023) compare LLMs’ performance on questions around entities with different frequencies. LLMs are found struggle to handle tail knowledge. However, the questions they study are different and can vary in difficulty levels, not only in entity frequencies. As Allen-Zhu and Li (2023) emphasized, we need a more “controlled, synthetic experiment that confirms the weakness of LLMs” nowadays. Therefore, existing comparisons are not enough since they can not guarantee that the low frequency is the only cause of LLMs’ poor performance.

To tackle the issue, we introduce our COMPARISONQA benchmark. Pairs of high-frequency and low-frequency entities share the same question, as shown in Figure 1. The shared abstract question, with a hypernym to represent the two specific entities, guarantees that the difference between such a pair is only the entity. This allows for a controllable comparison between high-frequency and low-frequency entities. COMPARISONQA is a large scale dataset containing 283K such question pairs. It is generated through an automatic pipeline base on raw knowledge base, ensuring both diversity and scalability. Through this benchmark, we can completely compare LLM’s performance on different knowledge frequencies.

![An example from COMPARISONQA](page_1016_370_355_355.png)

Figure 1: An example from COMPARISONQA

1https://github.com/HKUST-KnowComp/ComparisonQA
For a more robust and accurate evaluation, we use the multiple-choice format (Hendrycks et al., 2021; Geva et al., 2021) in our benchmark. But semantic shortcuts (Geirhos et al., 2020) between questions and options may help LLMs to guess the answer, which is also a common but severe problem recently. Thus, we further design a two-round method using both correctness and uncertainty to evaluate LLMs’ knowledge. During experiments, we found that LLMs have very poor robustness, especially on low-frequency knowledge, where even the powerful GPT-4o also performs badly.

Recent benchmarks, like SimpleQA (Wei et al., 2024a), ensure their difficulty by collecting questions adversarially against LLMs’ responses. But relying only on accuracy, they ignore the quality of their questions, and the difficulty is closely related to the models they use. Fortunately, our experiments find that uncertainty is also an effective tool in selecting both high-quality and shortcut-free questions while maintaining the benchmark size. Combining accuracy and uncertainty, we propose a new flexible method to select our subset called COMPARISONQA-Hard for future study. It contains 81K difficult low-frequency questions with high-quality and no semantic shortcuts.

In summary, we have three main contributions: (1) [Resource] We introduce COMPARISONQA benchmark, where a pair of entities share the same abstract question. It enables a more controllable and reasonable proof that LLMs perform worse when the required knowledge is less frequent. (§3) (2) [Method] We design a two-round method using correctness and uncertainty to evaluate LLMs’ robust knowledge. [Finding] LLMs can not stand such a test, especially on low-frequency knowledge, where even GPT-4o performs badly. (§4) (3) [Finding] Uncertainty is more helpful to find questions with high quality. [Resource] Through this, we select COMPARISONQA-Hard benchmark containing only hard and low-frequency questions of high quality and no shortcuts. (§5)

2 Related Works

2.1 Benchmarking LLMs’ Factuality

The factuality evaluation of LLMs has recently attracted significant attention. Some factuality benchmarks require open-ended generation by LLMs, such as SimpleQA (Wei et al., 2024a), Self-Aware (Yin et al., 2023), CLR-Fact (Zheng et al., 2024b), and HaluEval (Li et al., 2023). Such evaluations either rely heavily on expert annotation, or utilize automatic answer matching that sacrifices evaluation accuracy (Min et al., 2023; Chern et al., 2023; Wang et al., 2024b). Other benchmarks adopt the format of Yes-or-No questions (Geva et al., 2021; Zhang et al., 2024) or multiple-choice questions (MCQ) (Hendrycks et al., 2021; Wang et al., 2024a; Zheng et al., 2024a). These formats allow model responses to be easily parsed and compared with gold labels, enabling solid yet efficient evaluations.

2.2 Long-Tail Knowledge

Long-tail knowledge (Wei et al., 2024b; Chen et al., 2023) is an important aspect of factuality. Kumar et al. (2024) proposes an automatic approach to generate questions for tail entities. Kandpal et al. (2023) find LLMs struggle to learn long-tail knowledge. Other works study the influence of knowledge frequency: Mallen et al. (2023) introduced PopQA, a long-tail benchmark, and found that models’ performance will change with the frequency of entities in the questions. Sun et al. (2024) also proves this by constructing questions around head, torso, and tail entities. However, questions in these benchmarks are all in the form of open-ended generation, which can not be easily evaluated. They also depend on limited number of templates to produce QA questions from knowledge graphs, which will significantly harm the diversity of the benchmarks. Most importantly, the questions are different, so they may vary in difficulty levels, and thus can not provide fair comparisons.

2.3 Abstraction Knowledge

Existing works have studied various aspects of abstraction, for example, entity abstraction (Wu et al., 2012; Song et al., 2015; Xu et al., 2023), event abstraction (Wang et al., 2024d,c), and conceptual abstraction (Han et al., 2024). Abstraction has been shown to be beneficial for downstream tasks like commonsense reasoning, numerical reasoning, and logical reasoning (Zhou et al., 2024; Hong et al., 2024). In this paper, we control question difficulty by sharing the same abstraction form between a pair of entities.

3 COMPARISONQA

To ensure more fair and controllable comparisons between LLMs’ performance on high-frequency and low-frequency factual knowledge, we propose a new benchmark called COMPARISONQA.
Figure 2: An overview of our benchmark curation pipeline. It contains three parts. Through the first two parts, (1) Entity Pairs Extraction from DBpedia and (2) Abstract Question Generation, we can get the whole COMPARISONQA. And through the third part, (3) Hard High-Quality Question Selecting, we can get a harder subset, containing only difficult low-frequency questions with high quality and no semantic shortcut.

3.1 Question Formulation
For an accurate evaluation, our questions are in the form of multiple choice. Although recently there are several generative QA benchmarks, like SimpleQA (Wei et al., 2024a), and also some automatic methods (Zheng et al., 2023) to evaluate generated answers. There are still significant limitations in such generative QAs since the answer should be single and indisputable. Questions like What is the primary focus or intention behind Civil Procedure Rules cannot be included since there are many ways to answer this question. However, multiple-choice questions do not have such limitations.

Each piece of data in COMPARISONQA, shown in Figure 1, contains an abstract question shared by two entities having the same hypernym. One entity, having many relationships in DBpedia (Auer et al., 2007), which will be introduced next, is the high-frequency entity, and the other, with only a few relationships, is the low-frequency one. The question will have different answers for the two entities, respectively. Such data form can ensure detailed and controllable comparisons between entities with different frequencies, and that is why we call it COMPARISONQA. Containing 283K such question pairs, the benchmark is constructed through a fully automated pipeline, which is cheap and scalable.

3.2 Curation Pipeline
In this section, we discuss the data curation pipeline for our dataset. As shown in Figure 2, the pipeline contains three parts: (1) Entity Pairs Extraction from DBpedia, (2) Abstract Question Generation, and (3) Hard High-Quality Question Filtering. Here we will introduce the first two parts, used to build COMPARISONQA, and leave the third part, used to build COMPARISONQA-Hard, for §5.

3.2.1 Entity Pairs Extraction from DBpedia
Following Sun et al. (2024), an entity’s frequency is defined by its number of relationships in DBpedia. High-frequency entities are those whose cumulative relationships account for the first 1/3 of all sorted entities, and low-frequency entities are for the last 1/3. Details are shown in Appendix B.

In practice, we first get all the hypernyms in DBpedia and classify the entities belonging to them into high-frequent and low-frequent. Then, we map the entities one-by-one to get entity pairs. The reason behind this is that not every random pair of entities can have a shared abstract question easily. For example, it’s hard to write a shared question for Einstein and Apple even though they are all high-frequency entities. But it’s easy for two specific universities to share a same abstract question. In order for our entity pairs to produce high quality abstract questions, we ensure that the two entities have the same hypernym. This allows them to have similar descriptions, making it easier to generate an abstract question.

3.2.2 Abstract Question Generation
After acquiring the entity pairs, we adopt a multi-step curation pipeline to generate high-quality abstract multiple-choice questions: (1) Generate an abstract question (without options) according to the
DBpedia descriptions of both entity. (2) Separately generate the corresponding answers based on the descriptions, with length control to alleviate bias among candidate answers. (3) Generate distractors for both entities with length control. Compared to randomly selected distractors, LLM-selected distractors generation have demonstrated its effectiveness for the high relevance between distractors and designated choices (Zheng et al., 2024c). (4) Formulate the final multiple-choice question using the four answer candidates above. (5) Proofread the question according to the standards below.

The standards for questions are presented as follows: (1) Quality: The questions should have one and only one correct answer for both entities. (2) Semantic Shortcuts: The correct answer cannot be simply guessed by the names of the entities or the way the questions are asked. For example, the question: What was the primary operational location of Sydney O-Class Tram? A. Oslo, Norway B. Sydney, Australia C. Stockholm, Sweden D. Melbourne, Australia is not allowed since only the correct answer contains Sydney which is also in the entity’s name. (3) Length Bias: The four options in one question should have roughly the same length to avoid length bias. In summary, the generated questions should have high-quality and no shortcuts, in semantics or length.

During the curation, we utilize GPT-4o-mini (OpenAI, 2024a) with few-shot expert-written Chain-of-thought (CoT) demonstrations (Wei et al., 2022), with details provided in Appendix C.

3.3 Expert Verification

We enlist the help of three postgraduate students, each with extensive experience in NLP research, to validate the quality of these generated questions through a sample of 200 question pairs. The instruction is the same as the standard (1) given to LLMs above. The quality of each pair is decided by majority voting. Results show that their total agreement (all 3 experts have the same judgement) is 88.0%. And 95.5% of the abstract questions are considered correct and of high quality both for the high-frequency entity and the low-frequency entity, demonstrating the reliability of our benchmark.

3.4 Main Evaluations

COMPARISONQA is a large-scale benchmark comprising a total of 283,455 abstract questions, each paired with a high-frequency and a low-frequency instantiation. Detailed statistics are in Appendix A.

We experiment with a selection of LLMs on our COMPARISONQA benchmark to investigate their performance on high-frequency and low-frequency questions and also the difference between them.

3.4.1 Experiment Setup

Metric: We calculate Uncertainty, Accuracy, and Macro F1-score between model predictions and ground truth labels. We apply perplexity-based uncertainty for open source LLMs and verbalized uncertainty for proprietary LLMs due to several reasons, with details explained in Appendix D. Thus, we only compare uncertainty between high and low frequency within each setting separately.

Models: We experiment with 16 different models, with a full list in Appendix E, and categorize the evaluation into three types: (1) Open Source LLM Zero-Shot (Qin et al., 2023). (2) Open Source LLM Few-Shot (Brown et al., 2020). (3) Proprietary LLM API.

3.4.2 Results and Analysis

Evaluation results are reported in Table 1. Our observations include: (1) Huge drop in performance from high-freq to low-freq: All models suffer a performance decrease from high-frequency questions to low-frequency questions in all three settings. For instance, the accuracy and Macro F1-score of Llama-3-8B drop up to about 14 points in the few-shot setting. Proprietary models like GPT-4o are no exception. These all prove that LLM’s performance is closely related to the frequency of the knowledge in the corpus. (2) Increased Uncertainty from high-freq to low-freq: Similarly, the uncertainty of LLMs all increase from high-frequency questions to low-frequency questions. For example, the uncertainty difference of Gemma-2-9B is up to about 86 points when using zero-shot. The uncertainty in the zero-shot setting is generally higher than in the few-shot setting, and the difference is also more pronounced. We think it may be because LLMs find some familiar examples in the few-shot setting, which decreases their uncertainty. In spite of this, the difference is still clear between high-freq and low-freq. These all prove that LLMs not only perform better but also are more confident about high-frequency knowledge. (3) Few-shot helps a lot only for LLMs without instruction-tuning: Most non-instruction-tuned LLMs show a huge improvement in performance from zero-shot to few-shot, but performance is similar for those instruction-tuned LLMs. This could
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="3">High Freq Question</th>
    <th colspan="3">Low Freq Question</th>
    <th colspan="3">Average</th>
    <th rowspan="2">Difference (H → T)</th>
  </tr>
  <tr>
    <th>Unc.</th><th>Acc</th><th>Ma-F1</th>
    <th>Unc.</th><th>Acc</th><th>Ma-F1</th>
    <th>Unc.</th><th>Acc</th><th>Ma-F1</th>
    <th>Unc.</th><th>Acc</th><th>Ma-F1</th>
  </tr>
  <tr>
    <td>Random</td>
    <td>-</td><td>25.29</td><td>25.29</td>
    <td>-</td><td>25.22</td><td>25.22</td>
    <td>-</td><td>25.26</td><td>25.26</td>
    <td>-</td><td>0.07</td><td>0.07</td>
  </tr>
  <tr>
    <td>Majority</td>
    <td>-</td><td>25.70</td><td>10.22</td>
    <td>-</td><td>25.14</td><td>10.04</td>
    <td>-</td><td>25.42</td><td>10.13</td>
    <td>-</td><td>0.56</td><td>0.18</td>
  </tr>
</table>

LLM (Open Source) + Zero-Shot

<table>
  <tr>
    <th>Llama-3 sb</th><td>54.33</td><td>65.90</td><td>63.60</td><td>81.54</td><td>53.83</td><td>51.29</td><td>67.94</td><td>59.87</td><td>57.44</td><td>↑ 6.11</td><td>↓ 12.07</td><td>↓ 12.30</td>
  </tr>
  <tr>
    <th>Llama-3-Instruct sb</th><td>77.55</td><td>80.72</td><td>80.71</td><td>117.41</td><td>69.03</td><td>68.95</td><td>97.48</td><td>74.88</td><td>74.83</td><td>↑ 39.86</td><td>↓ 11.69</td><td>↓ 11.76</td>
  </tr>
  <tr>
    <th>Llama-3.1 sb</th><td>55.91</td><td>65.29</td><td>63.31</td><td>83.35</td><td>52.66</td><td>50.52</td><td>69.63</td><td>58.98</td><td>56.92</td><td>↑ 27.44</td><td>↓ 12.63</td><td>↓ 12.78</td>
  </tr>
  <tr>
    <th>Llama-3.1-Instruct sb</th><td>58.97</td><td>80.06</td><td>80.08</td><td>87.05</td><td>69.99</td><td>69.94</td><td>73.01</td><td>75.03</td><td>75.01</td><td>↑ 28.08</td><td>↓ 10.07</td><td>↓ 10.14</td>
  </tr>
  <tr>
    <th>Gemma-2 sb</th><td>124.97</td><td>64.50</td><td>64.80</td><td>211.34</td><td>52.24</td><td>51.23</td><td>168.16</td><td>58.37</td><td>58.01</td><td>↑ 86.37</td><td>↓ 12.26</td><td>↓ 13.57</td>
  </tr>
  <tr>
    <th>Phi-3.5-mini-Instruct 4B</th><td>27.81</td><td>72.81</td><td>72.78</td><td>39.80</td><td>65.26</td><td>65.00</td><td>33.81</td><td>69.04</td><td>68.89</td><td>↑ 11.99</td><td>↓ 7.55</td><td>↓ 7.78</td>
  </tr>
  <tr>
    <th>Falcon2 11B</th><td>56.93</td><td>70.72</td><td>69.80</td><td>87.31</td><td>58.07</td><td>56.70</td><td>72.12</td><td>64.40</td><td>63.25</td><td>↑ 30.38</td><td>↓ 12.65</td><td>↓ 13.10</td>
  </tr>
  <tr>
    <th>Mistral-v0.3 sb</th><td>39.83</td><td>65.55</td><td>63.11</td><td>56.99</td><td>53.36</td><td>50.26</td><td>48.41</td><td>59.46</td><td>56.69</td><td>↑ 17.16</td><td>↓ 12.19</td><td>↓ 12.85</td>
  </tr>
  <tr>
    <th>Mistral-v0.3-Instruct 7B</th><td>44.73</td><td>73.53</td><td>73.14</td><td>66.09</td><td>63.05</td><td>62.40</td><td>55.41</td><td>68.29</td><td>67.77</td><td>↑ 21.36</td><td>↓ 10.48</td><td>↓ 10.74</td>
  </tr>
</table>

LLM (Open Source) + Few-Shot

<table>
  <tr>
    <th>Llama-3 sb</th><td>21.89</td><td>75.57</td><td>75.55</td><td>23.75</td><td>61.00</td><td>61.01</td><td>22.82</td><td>68.29</td><td>68.28</td><td>↑ 1.86</td><td>↓ 14.57</td><td>↓ 14.55</td>
  </tr>
  <tr>
    <th>Llama-3-Instruct sb</th><td>26.20</td><td>79.94</td><td>79.92</td><td>28.70</td><td>67.98</td><td>67.95</td><td>27.45</td><td>73.96</td><td>73.93</td><td>↑ 2.50</td><td>↓ 11.96</td><td>↓ 11.96</td>
  </tr>
  <tr>
    <th>Llama-3.1 sb</th><td>20.82</td><td>74.91</td><td>74.89</td><td>22.62</td><td>62.00</td><td>62.01</td><td>21.72</td><td>68.46</td><td>68.45</td><td>↑ 1.81</td><td>↓ 12.91</td><td>↓ 12.90</td>
  </tr>
  <tr>
    <th>Llama-3.1-Instruct sb</th><td>20.63</td><td>79.74</td><td>79.74</td><td>22.48</td><td>69.09</td><td>69.07</td><td>21.56</td><td>74.42</td><td>74.40</td><td>↑ 1.85</td><td>↓ 10.65</td><td>↓ 10.66</td>
  </tr>
  <tr>
    <th>Gemma-2 sb</th><td>20.26</td><td>80.10</td><td>80.08</td><td>22.36</td><td>68.36</td><td>68.31</td><td>21.31</td><td>74.23</td><td>74.20</td><td>↑ 2.10</td><td>↓ 11.74</td><td>↓ 11.77</td>
  </tr>
  <tr>
    <th>Phi-3.5-mini-Instruct 4B</th><td>11.02</td><td>73.68</td><td>73.67</td><td>11.85</td><td>67.46</td><td>67.33</td><td>11.44</td><td>70.57</td><td>70.50</td><td>↑ 0.83</td><td>↓ 6.22</td><td>↓ 6.34</td>
  </tr>
  <tr>
    <th>Falcon2 11B</th><td>16.42</td><td>77.11</td><td>77.01</td><td>17.75</td><td>65.92</td><td>65.75</td><td>17.10</td><td>71.52</td><td>71.38</td><td>↑ 1.35</td><td>↓ 11.19</td><td>↓ 11.26</td>
  </tr>
  <tr>
    <th>Mistral-v0.3 sb</th><td>13.89</td><td>75.55</td><td>75.53</td><td>14.99</td><td>62.88</td><td>62.85</td><td>14.44</td><td>69.22</td><td>69.19</td><td>↑ 1.10</td><td>↓ 12.67</td><td>↓ 12.68</td>
  </tr>
  <tr>
    <th>Mistral-v0.3-Instruct 7B</th><td>15.97</td><td>74.46</td><td>74.40</td><td>17.40</td><td>65.51</td><td>65.35</td><td>16.69</td><td>69.99</td><td>69.87</td><td>↑ 1.43</td><td>↓ 8.95</td><td>↓ 9.05</td>
  </tr>
</table>

LLM (Proprietary) API

<table>
  <tr>
    <th>GPT4o-mini (Zero-Shot)</th><td>13.52</td><td>85.61</td><td>85.58</td><td>18.34</td><td>73.85</td><td>73.73</td><td>15.93</td><td>79.73</td><td>79.66</td><td>↑ 4.82</td><td>↓ 11.76</td><td>↓ 11.85</td>
  </tr>
  <tr>
    <th>GPT4o-mini (Few-Shot)</th><td>25.74</td><td>84.78</td><td>84.69</td><td>38.17</td><td>72.76</td><td>72.47</td><td>31.96</td><td>78.77</td><td>78.58</td><td>↑ 12.43</td><td>↓ 12.02</td><td>↓ 12.22</td>
  </tr>
  <tr>
    <th>GPT4o-mini (CoT)</th><td>10.53</td><td>86.25</td><td>86.25</td><td>12.27</td><td>74.39</td><td>74.40</td><td>11.40</td><td>80.32</td><td>80.32</td><td>↑ 1.74</td><td>↓ 11.85</td><td>↓ 11.85</td>
  </tr>
  <tr>
    <th>GPT4o (Zero-Shot)</th><td>14.18</td><td>93.86</td><td>93.95</td><td>30.98</td><td>85.76</td><td>86.69</td><td>22.58</td><td>89.81</td><td>90.32</td><td>↑ 16.80</td><td>↓ 8.10</td><td>↓ 7.26</td>
  </tr>
  <tr>
    <th>GPT4o (Few-Shot)</th><td>28.41</td><td>93.94</td><td>93.95</td><td>45.81</td><td>86.54</td><td>86.75</td><td>37.11</td><td>90.24</td><td>90.35</td><td>↑ 17.40</td><td>↓ 7.40</td><td>↓ 7.20</td>
  </tr>
  <tr>
    <th>GPT4o (CoT)</th><td>10.39</td><td>92.40</td><td>92.47</td><td>18.36</td><td>85.47</td><td>85.72</td><td>14.38</td><td>88.93</td><td>89.10</td><td>↑ 7.97</td><td>↓ 6.93</td><td>↓ 6.75</td>
  </tr>
</table>

Table 1: Performance of various LLMs on the testing set of COMPARISONQA. Unc., Acc, and Ma-F1 denote Uncertainty, Accuracy, and Macro F1-score. The Difference column shows how scores change from high-frequency questions to low-frequency questions. The best performances within each method are underlined, and the best among all methods are bold-faced. And for the Difference column, We underline the largest difference within each method and bold the one among all methods. More results can be seen in Table 10.

be because the few-shot examples only teach LLMs how to do multiple-choice questions, while those instruction-tuned ones have already learned. (4) CoT lowers uncertainty but does not awlays aid performance: It’s obvious that after the CoT inference, LLMs are more sure about their answers. However, results show that the average accuracy of GPT-4o even drops after adding CoT, which means CoT can not always help such factual questions.

4 Robust Knowledge Measurement

With the help of COMPARISONQA, we can conduct a more detailed and controllable study of the factual knowledge of LLMs.

When considering how humans tackle multiple-choice questions, it’s often the case that we do not really know the correct answers. Instead, we rely on semantic shortcuts within the questions to make educated guesses. Although we intentionally exclude these shortcuts when constructing the benchmark, they are difficult to eliminate entirely from multiple-choice questions. Such a situation often occurs even in human exam questions. Therefore, we need to devise an effective method to evaluate the robustness of factual knowledge of LLMs in the form of multiple-choice questions.

4.1 The Definition of Robust Knowledge

We categorize LLM’s results into four scenarios based on its uncertainty and the correctness of its answers. The uncertainty pertains to the statement based on pure Question+Answer (without options), which isolates the correct knowledge without the influence of the three distractors. Details are in Appendix D. Meanwhile, the correctness refers to the original multiple-choice question.

Low Uncertainty & Correct Answer: LLM shows confidence about the correct knowledge and also answers the question correctly. In this case, we consider the LLM to possess robust knowledge.

High Uncertainty & Incorrect Answer: LLM expresses uncertainty and answers incorrectly. So, we conclude that the LLM lacks the knowledge.
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="4">First Round</th>
    <th colspan="4">Second Round</th>
  </tr>
  <tr>
    <th>High</th>
    <th>Low</th>
    <th>Avg.</th>
    <th>Diff</th>
    <th>High</th>
    <th>Low</th>
    <th>Avg.</th>
    <th>Diff</th>
  </tr>
  <tr>
    <th colspan="9">Open Source LLM</th>
  </tr>
  <tr>
    <td>Llama-3 8B</td>
    <td>75.57</td>
    <td>61.00</td>
    <td>68.29</td>
    <td>14.57</td>
    <td>65.11 (-10.47)</td>
    <td>43.65 (-17.35)</td>
    <td>54.38 (-13.91)</td>
    <td>↓ 21.45 (+6.89)</td>
  </tr>
  <tr>
    <td>Llama-3-Instruct 8B</td>
    <td>79.94</td>
    <td>67.98</td>
    <td>73.96</td>
    <td>11.96</td>
    <td>68.13 (-11.82)</td>
    <td>48.73 (-19.26)</td>
    <td>58.43 (-15.54)</td>
    <td>↓ 19.40 (+7.44)</td>
  </tr>
  <tr>
    <td>Llama-3.1 8B</td>
    <td>74.91</td>
    <td>62.00</td>
    <td>68.46</td>
    <td>12.91</td>
    <td>65.66 (-9.25)</td>
    <td>45.21 (-16.78)</td>
    <td>55.44 (-13.02)</td>
    <td>↓ 20.45 (+7.53)</td>
  </tr>
  <tr>
    <td>Llama-3.1-Instruct 8B</td>
    <td>79.74</td>
    <td>69.09</td>
    <td>74.42</td>
    <td>10.65</td>
    <td>72.23 (-7.51)</td>
    <td>55.04 (-14.06)</td>
    <td>63.64 (-10.78)</td>
    <td>↓ 17.20 (+6.55)</td>
  </tr>
  <tr>
    <td>Llama-3.2 8B</td>
    <td>68.89</td>
    <td>57.26</td>
    <td>63.08</td>
    <td>11.63</td>
    <td>58.89 (-10.00)</td>
    <td>41.11 (-16.15)</td>
    <td>50.00 (-13.07)</td>
    <td>↓ 17.78 (+6.15)</td>
  </tr>
  <tr>
    <td>Llama-3.2-Instruct 8B</td>
    <td>71.43</td>
    <td>62.12</td>
    <td>66.78</td>
    <td>9.32</td>
    <td>64.07 (-7.37)</td>
    <td>46.68 (-15.44)</td>
    <td>55.37 (-11.40)</td>
    <td>↓ 17.39 (+8.07)</td>
  </tr>
  <tr>
    <td>Gemma-2 2B</td>
    <td>62.99</td>
    <td>50.93</td>
    <td>56.96</td>
    <td>12.06</td>
    <td>52.54 (-10.44)</td>
    <td>35.40 (-15.53)</td>
    <td>43.97 (-12.99)</td>
    <td>↓ 17.14 (+5.09)</td>
  </tr>
  <tr>
    <td>Gemma-2 9B</td>
    <td>80.10</td>
    <td>68.36</td>
    <td>74.23</td>
    <td>11.74</td>
    <td>71.39 (-8.71)</td>
    <td>50.48 (-17.88)</td>
    <td>60.94 (-13.30)</td>
    <td>↓ 20.92 (+9.18)</td>
  </tr>
  <tr>
    <td>Phi-3.5-mini 4B</td>
    <td>73.68</td>
    <td>67.46</td>
    <td>70.57</td>
    <td>6.22</td>
    <td>66.82 (-6.86)</td>
    <td>53.51 (-13.95)</td>
    <td>60.16 (-10.41)</td>
    <td>↓ 13.31 (+7.08)</td>
  </tr>
  <tr>
    <td>Falcon2 11B</td>
    <td>77.11</td>
    <td>65.92</td>
    <td>71.52</td>
    <td>11.19</td>
    <td>65.17 (-11.94)</td>
    <td>46.88 (-19.04)</td>
    <td>56.02 (-15.49)</td>
    <td>↓ 18.29 (+7.10)</td>
  </tr>
  <tr>
    <td>Mistral-v0.3 7B</td>
    <td>75.55</td>
    <td>62.88</td>
    <td>69.22</td>
    <td>12.67</td>
    <td>67.86 (7.69)</td>
    <td>49.05 (-13.83)</td>
    <td>58.46 (-10.76)</td>
    <td>↓ 18.82 (+6.15)</td>
  </tr>
  <tr>
    <td>Mistral-v0.3-Instruct 7B</td>
    <td>74.46</td>
    <td>65.51</td>
    <td>69.99</td>
    <td>8.95</td>
    <td>67.29 (-7.17)</td>
    <td>51.69 (-13.82)</td>
    <td>59.49 (-10.50)</td>
    <td>↓ 15.60 (+6.65)</td>
  </tr>
  <tr>
    <th colspan="9">Proprietary LLM</th>
  </tr>
  <tr>
    <td>GPT4o-mini</td>
    <td>84.78</td>
    <td>72.76</td>
    <td>78.77</td>
    <td>12.02</td>
    <td>71.00 (-13.78)</td>
    <td>38.25 (-34.51)</td>
    <td>54.63 (-24.14)</td>
    <td>↓ 32.75 (+20.73)</td>
  </tr>
  <tr>
    <td>GPT4o</td>
    <td><b>93.94</b></td>
    <td><b>86.54</b></td>
    <td><b>90.24</b></td>
    <td>7.40</td>
    <td><b>79.85</b> (-14.09)</td>
    <td><b>55.94</b> (-30.6)</td>
    <td><b>67.90</b> (-22.34)</td>
    <td>↓ 23.91 (+16.51)</td>
  </tr>
</table>

Table 2: Accuracy scores in LLMs’ robust knowledge measurement. We also report the changes in the scores from the first round to the second round. The best performances within each method are underlined, and the best among all methods are bold-faced. And for the Difference column and values in parentheses, We underline the largest difference within each method and bold the one among all methods.

High Uncertainty & Correct Answer: LLM is unsure but answers correctly. This indicates that it may retain the correct knowledge, but the memory is vague, or that the semantic shortcuts in the question lead to the correct answer.

Low Uncertainty & Incorrect Answer: LLM is confident yet answers incorrectly. This could result from the LLM recalling incorrect knowledge or from misleading distractors in questions.

In the first two categories, we can determine whether the LLM truly possesses the knowledge. However, in the latter two cases, multiple factors influence the final results, and the LLM’s grasp of knowledge is not robust.

This classification method leverages the strengths of both multiple-choice and generative questions, since we collect the uncertainty score without the distractors. While multiple-choice questions are easy to evaluate, they may allow for shortcuts; generative questions, on the other hand, are the opposite. Our method capitalizes on the uncertainty inherent in generative questions, which do not have shortcuts, and the accuracy of easily parsed answers provided by multiple-choice questions. This approach ensures that evaluation remains straightforward while fully addressing the potential for shortcuts.

4.2 The Two-Round Measurement

We introduce our two-round measurement, which can be applied to any multiple-choice benchmarks, based on the four categories. In the first round, we present multiple-choice questions to evaluate LLM’s performance. Then, results are classified into four categories. For questions falling into the latter two cases (high-uncertainty correct & low-uncertainty incorrect), we will conduct a second round questioning. Scores will be modified if the correctness of any questions changes in this round.

For questions requiring reassessment in the second round, we ask LLMs to judge whether the four statements, with details in Appendix D, are true or false. The LLM is considered to truly possess the knowledge only when all four statements are accurately judged. On one hand, the second round provides an opportunity for LLMs to correct the answer by breaking distractors into separate questions, and on the other hand, it identifies questions where the LLMs simply guess the correct answers.

Compared to breaking down the multiple-choice questions into four correctness judgments directly from the start, our two-round approach offers a comprehensive analysis of how LLMs’ performance changes from the first to the second stage. It leverages both uncertainty and correctness, providing deeper insights into LLMs’ confidence and robustness regarding the factual knowledge they retain. Additionally, this method enables us to determine whether a particular result is due to LLM’s lack of knowledge or the shortcuts and misleading distractors in the multiple-choice questions.

Although the second round of evaluation is stricter, our ComparisonQA benchmark ensures fair classification and comparison between high-
Figure 3: Heatmaps illustrating how subset quality changes with incorrect model number and high uncertainty remaining ratio. The former refers to the minimum number of times that each remaining question in the subset is answered incorrectly. The latter refers to the proportion of high-uncertainty questions that are retained in the subset.

frequency and low-frequency questions under the same setting. Because we have a shared abstract question for each pair of entities, controlling other factors which may affect LLM’s uncertainty and accuracy. Thus, we can rule out all the other factors to purely compare the difference between high-frequency and low-frequency knowledge.

4.3 Experiments and Analysis

Here, we conduct the experiments using this method to measure LLMs’ knowledge robustness in a few-shot manner. Results are shown in Table 2

There are very interesting observations from the results: (1) GPT-4o and GPT-4o-mini can not stand the test of robust knowledge measurement especially on low-frequency knowledge: Their performance is still very good on high-frequency questions after the second round but drops a lot on low-frequency ones. The accuracy of GPT-4o-mini on low-frequency questions after the second round is only 38.25, about 35 points lower than the first round. Their average performance even drop the most among all the evaluated models. (2) The LLMs’ grasp of low-frequency knowledge is less robust than that of high-frequency knowledge: Performance of LLMs in the second round all drop much more on low-frequency questions than on high-frequency questions. For example, in all of the open source LLMs, the accuracy of Llama-3.1-SB-Instruct in the second round, whose average performance is the best in the first round, dropped up to about 25 points on low-frequency questions while only about 7 points on high-frequency questions. (3) LLMs which can stand the test of robust knowledge measurement must have a very accurate grasp of low-frequency knowledge: LLMs whose average accuracy in the second round is higher than 55 are all those who performed relatively better on the low-frequency questions. This illustrates the importance of the LLMs mastering low-frequency knowledge if they want to be reliable models in the aspect of factual knowledge.

5 COMPARISONQA-Hard

Directly collected questions, especially the high-frequency part, are a bit simple for today’s LLMs. More importantly, they may have low quality and semantic shortcuts. So we introduce a new method to select a subset called COMPARISONQA-Hard, containing only difficult, low-frequency questions that have high quality and no semantic shortcuts.

5.1 Hard High-Quality Question Filtering

The filtering method utilizes both correctness and uncertainty, similar to the last section. Previous benchmarks, like SimpleQA (Wei et al., 2024a) and MMLU-Pro (Wang et al., 2024a), were collected adversarially based on LLMs’ responses to ensure question difficulty. We enhance this method by also considering LLMs’ uncertainty to achieve the selection of high-quality, shortcut-free questions.

As illustrated in the third part of Figure 2, we collect the correctness and uncertainty from six open-source LLMs, listed in Appendix E, for questions with low-frequency entities. Different from above, both metrics here are about the entire multiple-choice questions, considering all the four options’ quality. We choose the questions that many models answer incorrectly and exhibit high uncertainty for our hard subset. We define two hyperparameters: incorrect model number and high uncertainty remaining ratio. The former is the minimum number of times that each remaining question is answered incorrectly. The latter is the proportion of high-uncertainty questions (sort by the sum of the uncertainty of all models) retained in the subset.

The method is designed based on this assumption: Low-quality questions and those with semantic shortcuts are often associated with correct an-
<table>
  <tr>
    <th>Method</th>
    <th>Backbone</th>
    <th>Unc.</th>
    <th>Acc.</th>
    <th>Ma-F1</th>
  </tr>
  <tr>
    <td rowspan="8">Open Source LLM</td>
    <td>Llama-2 sb</td>
    <td>25.04</td>
    <td>27.95</td>
    <td>27.74</td>
  </tr>
  <tr>
    <td>Llama-3-Instruct sb</td>
    <td>30.57</td>
    <td>22.38</td>
    <td>22.32</td>
  </tr>
  <tr>
    <td>Llama-3.1 sb</td>
    <td>23.87</td>
    <td>28.76</td>
    <td>28.40</td>
  </tr>
  <tr>
    <td>Llama-3.1-Instruct sb</td>
    <td>23.77</td>
    <td>27.06</td>
    <td>26.59</td>
  </tr>
  <tr>
    <td>Llama-3.2 sb</td>
    <td>26.97</td>
    <td>27.10</td>
    <td>27.05</td>
  </tr>
  <tr>
    <td>Llama-3.2-Instruct sb</td>
    <td>27.04</td>
    <td>28.68</td>
    <td>28.65</td>
  </tr>
  <tr>
    <td>Gemma-2 zb</td>
    <td>31.07</td>
    <td>22.80</td>
    <td>22.35</td>
  </tr>
  <tr>
    <td>Gemma-2 sb</td>
    <td>23.67</td>
    <td>24.71</td>
    <td>24.64</td>
  </tr>
  <tr>
    <td>Phi-3.5-mini 4b</td>
    <td><b>12.42</b></td>
    <td>31.87</td>
    <td>31.19</td>
  </tr>
  <tr>
    <td>Falcon2 11B</td>
    <td>18.65</td>
    <td>23.70</td>
    <td>23.64</td>
  </tr>
  <tr>
    <td>Mistral-v0.3 zb</td>
    <td>15.69</td>
    <td>23.96</td>
    <td>23.74</td>
  </tr>
  <tr>
    <td>Mistral-v0.3-Instruct zb</td>
    <td>18.36</td>
    <td>23.65</td>
    <td>23.25</td>
  </tr>
  <tr>
    <td rowspan="2">Proprietary LLM</td>
    <td>GPT-4o-mini</td>
    <td>46.49</td>
    <td>38.13</td>
    <td>37.82</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>54.66</td>
    <td><b>69.98</b></td>
    <td><b>70.02</b></td>
  </tr>
</table>

Table 3: Performance of various LLMs on the testing set of COMPARISONQA-Hard. Unc., Acc., and Ma-F1, denote Uncertainty, Accuracy, and Macro F1-score. The best performances within each method are underlined and the best among all methods are bold-faced.

swers and low uncertainty across different models, as our benchmark is constructed by LLMs-generated questions. The rationale behind this is as follows: (1) For low-quality questions, if the question or answer is incorrect, it is likely that the LLM did not generate it from the descriptions but rather from its internal knowledge. Consequently, models may display high confidence and yield correct answers. (2) For questions containing shortcuts, models may cheat through shortcuts, resulting in lower uncertainty and higher accuracy. These will all be proved in the following experiments.

5.2 Parameters Chosen by Expert Verification
To validate our filtering method, we invite experts to annotate the shortcuts in the 200 randomly sampled questions in the same setting mentioned in §3.3. The instruction is the same as the standard (2) given to LLMs above. Results show that 9.4% of the 95.5% correct and high quality questions are identified as having semantic shortcuts.

Then, we examine how the following metrics change with different settings regarding correctness and uncertainty, which are illustrated in Figure 3:
(1) Remaining Low-Quality Ratio (the proportion of remaining low-quality problems relative to the original number of low-quality problems),
(2) Remaining Shortcut Ratio (the proportion of remaining problems with shortcuts relative to the original number of problems with shortcuts), (3) GPT-4o Accuracy (the accuracy of GPT-4o on the remaining questions), and (4) Remaining Question Count (the size of the remaining questions). The results suggest that both uncertainty and correctness contribute to the selection of high-quality, shortcut-free questions, thereby demonstrating the effectiveness of our method. While it is expected that different models’ correctness would aid in identifying more challenging questions, it is noteworthy that uncertainty proved to be more effective in selecting high-quality and shortcut-free questions while maintaining the dataset size, validating the inclusion of uncertainty in our filtering method.

Finally, we choose to set incorrect model number to 3 and high uncertainty remaining ratio to 0.8 according to Figure 3. It is a trade-off between quality, difficulty, and subset size. In this setting, only 1.5% of the total questions are of low quality, and 2.1% with shortcuts. Finally, the subset size is 81K, with a GPT-4o accuracy of 70%. Detailed statistics are shown in Appendix A.

5.3 Experiments and Analysis
Then we conduct experiments on COMPARISONQA-Hard in a few-shot manner, with results shown in Table 3. In the multiple-choice question format, the open-source LLMs all significantly underperform, indicating the difficulty of our benchmark. For the proprietary LLMs, GPT-4o-mini also has a poor performance with an accuracy of about 38, even though we do not use it when constructing the subset. GPT-4o is better with an accuracy of about 70, but still has a huge room for future enhancement. And, predictably, its knowledge robustness will drop much more on this benchmark according to our experiments in §4.3.

6 Conclusions
In this paper, we first introduce COMPARISONQA benchmark to evaluate LLMs’ factual knowledge, with a fully automatic pipeline. This benchmark allows for more controllable and detailed comparisons between high-frequency and low-frequency knowledge of LLMs. Then, we propose a two-round method utilizing correctness and uncertainty to measure LLMs’ knowledge robustness. And we are surprised to find that even powerful LLMs like GPT-4o can not stand such a test, especially on the low-frequency knowledge. At last, we discover that uncertainty is more effective in filtering out questions with low quality and shortcuts compared with correctness, which is often used by recent works. Based on this method, we provide a subset called COMPARISONQA-Hard, which contains only difficult low-frequency questions of high quality and no shortcuts for future study.
Limitations

While we contribute valuable resources, methods, and findings to advance the probing of LLMs’ factual knowledge, several limitations still exist that cannot be covered in this single work.

In this paper, we provide a shared abstract question with the entities being the only varying part. However, due to the sharing of such abstract questions, it is difficult to contain more entities with different frequencies in the same question because many entities lack sufficient shared features to generate common questions.

Our approach ensures that the difference between a pair is only the entity. Future research could investigate methods for measuring the entire knowledge frequency required to solve the questions, rather than limiting it to entity frequency. We believe it is a challenging but valuable task.

Additionally, our focus is on fixed knowledge, but fast-changing factual knowledge (Do et al., 2024b) and other knowledge (Do et al., 2024a; Wang et al., 2022, 2023) also deserves attention. Specifically, exploring how knowledge frequency can assist LLMs in acquiring new information (Choi et al., 2023; Zong et al., 2023) is a worthwhile area for further study.

Ethics Statement

Offensive Content Elimination. Our benchmark curation pipeline, which involves generating content using LLMs, requires stringent measures to ensure that generated responses are free from offensive material. We manually review a random sample of 200 data instances from COMPARISONQA for any offensive content. Based on our annotations, we have not detected any offensive content. Therefore, we believe our dataset is safe and will not yield any negative societal impact.

Licenses. We will share our code under the MIT license, allowing other researchers free access to our resources for research purposes. Our dataset will be released under a CC license, also providing scholars with free access. We take full responsibility for any rights violations or issues related to the data license. The DBpedia dataset used in this paper is shared under the CC BY-SA license, permitting its use for research. As for language models, we access all open-source LMs via the Huggingface Hub (Wolf et al., 2020). All associated licenses permit user access for research purposes, and we have agreed to adhere to all terms of use.

Annotations. For expert verifications, we have obtained IRB approval and support from our institution’s department, enabling us to invite expert graduate students to validate the quality of our data. They all agree to participate voluntarily without being compensated. We have made significant efforts to eliminate offensive content, thereby ensuring that no annotators are offended.

Acknowledgments

The authors of this paper were supported by the ITSP Platform Research Project (ITS/189/23FP) from ITC of Hong Kong, SAR, China, and the AoE (AoE/E-601/24-N), the RIF (R6021-20) and the GRF (16205322) from RGC of Hong Kong,SAR, China.

References

Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat S. Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Amit Garg, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademli, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Chen Liang, Weishung Liu, Eric Lin, Zeqi Lin, Piyush Madan, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Xia Song, Masahiro Tanaka, Xin Wang, Rachel Ward, Guanhua Wang, Philipp Witte, Michael Wyatt, Can Xu, Jiahang Xu, Sonali Yadav, Fan Yang, Ziyi Yang, Donghan Yu, Chengruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. 2024. Phi-3 technical report: A highly capable language model locally on your phone. CoRR, abs/2404.14219.

Zeyuan Allen-Zhu and Yuanzhi Li. 2023. Physics of language models: Part 3.2, knowledge manipulation. CoRR, abs/2309.14402.

Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary G. Ives. 2007. Dbpedia: A nucleus for a web of open data. In The Semantic Web, 6th International Semantic Web Conference, 2nd Asian Semantic Web Conference,
ISWC 2007 + ASWC 2007, Busan, Korea, November 11-15, 2007, volume 4825 of Lecture Notes in Computer Science, pages 722–735. Springer.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.

Lihu Chen, Simon Razniewski, and Gerhard Weikum. 2023. Knowledge base completion for long-tail entities. CoRR, abs/2306.17472.

I-Chun Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, and Pengfei Liu. 2023. Factool: Factuality detection in generative AI - A tool augmented framework for multi-task and multi-domain scenarios. CoRR, abs/2307.13528.

Sehyun Choi, Tianqing Fang, Zhaowei Wang, and Yangqiu Song. 2023. KCTS: knowledge-constrained tree search decoding with token-level hallucination detection. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 14035–14053. Association for Computational Linguistics.

Quyet V. Do, Tianqing Fang, Shizhe Diao, Zhaowei Wang, and Yangqiu Song. 2024a. Constraintchecker: A plugin for large language models to reason on commonsense knowledge bases. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2024 - Volume 1: Long Papers, St. Julian’s, Malta, March 17-22, 2024, pages 714–731. Association for Computational Linguistics.

Quyet V Do, Junze Li, Tung-Duong Vuong, Zhaowei Wang, Yangqiu Song, and Xiaojuan Ma. 2024b. What really is commonsense knowledge? arXiv preprint arXiv:2411.03964.

Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, AoBo Yang, Archi Mitra, Archie SravanKumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Alonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esibou, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Milon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Jun-teng Jia, Kalyan Vasudean Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, and Kevin Stone. 2024. The llama 3 herd of models. CoRR, abs/2407.21783.

Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard S. Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. 2020. Shortcut learning in deep neural networks. Nat. Mach. Intell., 2(11):665–673.

Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? A question answering benchmark with implicit reasoning strategies. Trans. Assoc. Comput. Linguistics, 9:346–361.

Kaiqiao Han, Tianqing Fang, Zhaowei Wang, Yangqiu Song, and Mark Steedman. 2024. Concept-reversed winograd schema challenge: Evaluating and improving robust reasoning in large language models via abstraction. CoRR, abs/2410.12040.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.

Ruxin Hong, Hongming Zhang, Xiaoman Pan, Dong Yu, and Changshui Zhang. 2024. Abstraction-of-thought makes language models better reasoners. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pages 1993–2027. Association for Computational Linguistics.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Bernard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. CoRR, abs/2310.06825.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 15696–15707. PMLR.

Rohan Kumar, Youngmin Kim, Sunitha Ravi, Haitian Sun, Christos Faloutsos, Ruslan Salakhutdinov, and Minji Yoon. 2024. Automatic question-answer generation for long-tail knowledge. CoRR, abs/2403.01382.

Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023. Haluveal: A large-scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 6449–6464. Association for Computational Linguistics.

Jiacheng Liu, Wenya Wang, Dianzhuo Wang, Noah A. Smith, Yejin Choi, and Hannaneh Hajishirzi. 2023. Vera: A general-purpose plausibility estimation model for commonsense statements. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 1264–1287. Association for Computational Linguistics.

Quentin Malartic, Nilabhra Roy Chowdhury, Ruxandra Cojocaru, Mugariya Farooq, Giulia Campesan, Yasser Abdelaziz Dahou Djilali, Sanath Narayan, Ankit Singh, Maksim Velikanov, Basma El Amel Boussaha, Mohammed Al-Yafeai, Hamza Alobeidli, Leen Al Qadi, Mohamed El Amine Seddik, Kirill Fedyanin, Réda Alami, and Hakim Hacid. 2024. Falcon-2-11b technical report. CoRR, abs/2407.14885.

Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 9802–9822. Association for Computational Linguistics.

Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Husseinot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Cristian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, and et al. 2024. Gemma: Open models based on gemini research and technology. CoRR, abs/2403.08295.

Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 12076–12100. Association for Computational Linguistics.

OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774.

OpenAI. 2024a. Gpt-4o mini: advancing cost-efficient intelligence. OpenAI.

OpenAI. 2024b. Hello gpt-4o. OpenAI.

Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general-purpose natural language processing task solver? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 1339–1384. Association for Computational Linguistics.

Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Husseinot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Patterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozińska, Dustin Herbisson, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgeni Elyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimeczak-Plucińska, Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjøesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus,
Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahlhinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardesher Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara McCarthy, Sarah Perrin, Sébastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruthi Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Wooahun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitoa Gong, Zichuan Wei, Victor Cotrutz, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. 2024. Gemma 2: Improving open language models at a practical size. CoRR, abs/2408.00118.

Yangqiu Song, Shusen Wang, and Haixun Wang. 2015. Open domain short text conceptualization: A generative + descriptive modeling approach. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015, pages 3820–3826. AAAI Press.

Kai Sun, Yifan Ethan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. 2024. Head-to-tail: How knowledgeable are large language models (llms)? A.K.A. will llms replace knowledge graphs? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pages 311–325. Association for Computational Linguistics.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esibou, Jude Fernandes, Jeremy Fu, Wenying Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharar Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288.

Roman Vashurin, Ekaterina Fadeeva, Artem Vazhenstev, Akim Tsvigun, Daniil Vasilev, Rui Xing, Abdelrahman Boda Sadallah, Lyudmila Rvanova, Sergey Petrakov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov, and Artem Shelmanov. 2024. Benchmarking uncertainty quantification methods for large language models with Impolygraph. CoRR, abs/2406.15627.

Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhui Chen. 2024a. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. CoRR, abs/2406.01574.

Yuxia Wang, Revanth Gangi Reddy, Zain Muhammad Mujahid, Arnav Arora, Aleksandr Rubashevskii, Jiuhui Geng, Osama Mohammed Afzal, Liangming Pan, Nadav Borenstein, Aditya Pillai, Isabelle Augenstein, Iryna Gurevych, and Preslav Nakov. 2024b. Factcheck-bench: Fine-grained evaluation benchmark for automatic fact-checkers. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pages 14199–14230. Association for Computational Linguistics.

Zhaowei Wang, Quyet V. Do, Hongming Zhang, Jiayao Zhang, Weiqi Wang, Tianqing Fang, Yangqiu Song, Ginny Y. Wong, and Simon See. 2023. COLA: contextualized commonsense causal reasoning from the causal inference perspective. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). ACL 2023, Toronto, Canada, July 9-14, 2023, pages 5253–5271. Association for Computational Linguistics.

Zhaowei Wang, Wei Fan, Qing Zong, Hongming Zhang, Sehyun Choi, Tianqing Fang, Xin Liu, Yangqiu Song, Ginny Y. Wong, and Simon See. 2024c. Absinstruct: Eliciting abstraction ability from llms through explanation tuning with plausibility estimation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 973–994. Association for Computational Linguistics.

Zhaowei Wang, Haochen Shi, Weiqi Wang, Tianqing Fang, Hongming Zhang, Sehyun Choi, Xin Liu, and Yangqiu Song. 2024d. Abspyramid: Benchmarking
the abstraction ability of language models with a unified entailment graph. In Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pages 3991–4010. Association for Computational Linguistics.

Zhaowei Wang, Hongming Zhang, Tianqing Fang, Yangqiu Song, Ginny Y. Wong, and Simon See. 2022. Subeventwriter: Iterative sub-event sequence generation with coherence controller. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 1590–1604. Association for Computational Linguistics.

Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. 2024a. Measuring short-form factuality in large language models.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.

Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, and Quoc V. Le. 2024b. Long-form factuality in large language models. CoRR, abs/2403.18802.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, EMNLP 2020 - Demos, Online, November 16-20, 2020, pages 38–45. Association for Computational Linguistics.

Wentao Wu, Hongsong Li, Haixun Wang, and Kenny Qili Zhu. 2012. Probase: a probabilistic taxonomy for text understanding. In Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2012, Scottsdale, AZ, USA, May 20-24, 2012, pages 481–492. ACM.

Zhenran Xu, Yulin Chen, and Baotian Hu. 2023. Improving biomedical entity linking with cross-entity interaction. In Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pages 13869–13877. AAAI Press.

Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. 2023. Do large language models know what they don’t know? In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 8653–8665. Association for Computational Linguistics.

Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A. Smith. 2024. How language model hallucinations can snowball. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.

Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024a. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

Tianshi Zheng, Jiaxin Bai, Yicheng Wang, Tianqing Fang, Yue Guo, Yauwai Yim, and Yangqiu Song. 2024b. Clr-fact: Evaluating the complex logical reasoning capability of large language models over factual knowledge. Preprint, arXiv:2407.20564.

Tianshi Zheng, Weihan Li, Jiaxin Bai, Weiqi Wang, and Yangqiu Song. 2024c. Assessing the robustness of retrieval-augmented generation systems in k-12 educational question answering with knowledge discrepancies. Preprint, arXiv:2412.08985.

Ben Zhou, Hongming Zhang, Sihao Chen, Dian Yu, Hongwei Wang, Baolin Peng, Dan Roth, and Dong Yu. 2024. Conceptual and unbiased reasoning in language models. CoRR, abs/2404.00205.

Qing Zong, Zhaowei Wang, Baixuan Xu, Tianshi Zheng, Haochen Shi, Weiqi Wang, Yangqiu Song, Ginny Y. Wong, and Simon See. 2023. TILFA: A unified framework for text, image, and layout fusion in argument mining. In Proceedings of the 10th Workshop on Argument Mining, ArgMining 2023, Singapore, December 7, 2023, pages 139–147. Association for Computational Linguistics.
Appendices

A Benchmark Statistics

COMPARISONQA is a large-scale benchmark comprising a total of 283,455 abstract question pairs, each paired with a high-frequency and a low-frequency entity. We guarantee that each entity corresponds to a single question sourced from 9166 hypernyms, to ensure no overlap. We partition our data into training, validation, and testing splits following an 8:1:1 ratio, ensuring that entities of different frequency intervals are evenly distributed in each split. Specific details are shown in Table 4.

And for COMPARISONQA-Hard, there are 81,136 high-quality and shortcut-free questions with low-frequency entities from 4,876 hypernyms in total, with details shown in Table 5.

<table>
  <tr>
    <th>Split</th>
    <th>#Q. Pair</th>
    <th>#Entity</th>
    <th>#Hyper.</th>
    <th>#H. R.</th>
    <th>#L. R.</th>
  </tr>
  <tr>
    <td>Train</td>
    <td>226,762</td>
    <td>453,524</td>
    <td>8,430</td>
    <td>628</td>
    <td>76</td>
  </tr>
  <tr>
    <td>Valid</td>
    <td>28,345</td>
    <td>56,690</td>
    <td>3,316</td>
    <td>627</td>
    <td>76</td>
  </tr>
  <tr>
    <td>Test</td>
    <td>28,348</td>
    <td>56,696</td>
    <td>3,314</td>
    <td>629</td>
    <td>76</td>
  </tr>
  <tr>
    <td>Total</td>
    <td>283,455</td>
    <td>566,910</td>
    <td>9,166</td>
    <td>628</td>
    <td>76</td>
  </tr>
</table>

Table 4: The statistics of COMPARISONQA benchmark. #Q. Pair refers to the number of question pairs. Hyper means hypernym. #H. R. and #L. R. refer to the average number of relationships of high-frequency entities and low-frequency entities respectively.

<table>
  <tr>
    <th>Split</th>
    <th>#Q. = #Entity</th>
    <th>#Hyper.</th>
    <th>#L. R.</th>
  </tr>
  <tr>
    <td>Train</td>
    <td>65,057</td>
    <td>4,396</td>
    <td>77.45</td>
  </tr>
  <tr>
    <td>Valid</td>
    <td>7,978</td>
    <td>1,466</td>
    <td>77.23</td>
  </tr>
  <tr>
    <td>Test</td>
    <td>8,101</td>
    <td>1,484</td>
    <td>77.65</td>
  </tr>
  <tr>
    <td>Total</td>
    <td>81,136</td>
    <td>4,876</td>
    <td>77.44</td>
  </tr>
</table>

Table 5: The statistics of COMPARISONQA-Hard benchmark. Here the number of questions is equal to the number of entities since it only has low frequency entities.

For benchmark quality, we further compute the correctness of the annotated date in §3.3 coming from the train, validation, and test set separately. The results presented in Table 6 show the high correctness in all splits of our benchmark. Besides, our labeling is very strict. If either of the questions in one pair is wrong, we will judge the whole question pair incorrect. Since each pair contains two questions, actually only about 2.25% of the total questions are incorrect.

<table>
  <tr>
    <th>Split</th>
    <th>Correctness</th>
    <th>Agreement</th>
  </tr>
  <tr>
    <td>Train</td>
    <td>95.68</td>
    <td>88.27</td>
  </tr>
  <tr>
    <td>Valid</td>
    <td>93.75</td>
    <td>87.50</td>
  </tr>
  <tr>
    <td>Test</td>
    <td>95.45</td>
    <td>90.91</td>
  </tr>
</table>

Table 6: The statistics of COMPARISONQA-Hard benchmark. Here the number of questions is equal to the number of entities since it only has low frequency entities.

B Definition of High and Low Frequency

First, we randomly sample 1K entities from DBpedia and compute their relationship count separately. This process is to study the distribution of entities’ relationship counts in DBpedia and find boundaries for high-frequency and low-frequency entities. Then we sort these entities in order of their relationship count from highest to lowest. High-frequency entities possess cumulative relationship count of up to 1/3 of all entities, while low frequency entities range from 2/3 to 1. We exclude those between 1/3 and 2/3 to make comparison clear.

Repeating this process 3 times, we get the number of relationships to distinguish high-frequency and low-frequency entities, which are higher than 185 and lower than 107 respectively. (Of the 1K randomly sampled entities, 119 entities are identified as high frequent and 621 entities as low frequent.) These two numbers are then used to classify all the high-frequency and low-frequency entities.

It is necessary to randomly select 1K entities first to calculate the two boundaries, since the time to compute the cumulative relationship count for all the sorted entities in DBpedia is unaffordable.

C Details in COMPARISONQA Construction

In the process of entity pairs extraction, both hypernyms that do not have enough entities and low-frequency entities that do not have enough high-frequency entities to pair with are all discarded. Finally, we get 293K entities pairs from 9,261 hypernyms in total.

These pairs are then fed into the LLM to generate questions. Each pair has a shared abstract question respectively. Since a hypernym can have several entity pairs, the final number of abstract questions, which is equal to the number of pairs, is more than that of hypernyms. After LLM’s proofread stage, there are 283K abstract questions left, which are used to build our COMPARISONQA benchmark. Entities belonging to these questions come from
<table>
  <tr>
    <th>Task</th>
    <th>Prompt</th>
  </tr>
  <tr>
    <td>Question Generation</td>
    <td>You will be given two entities belonging to the same hypernym. Generate a shared multiple choice question for both entities based on their descriptions according to the following 5 steps.<br>Requirements: First, the questions should have one and only one correct answer for both entities. Second, the correct answer cannot be simply guessed by the names of the entities or the way the questions are asked. For example, the question: "What was the primary operational location of Sydney O-Class Tram? A. Oslo, Norway B. Sydney, Australia C. Stockholm, Sweden D. Melbourne, Australia" is not allowed since only the correct answer contains "Sydney" which is also in the entity's name. Third, the four options should have roughly the same length.<br>Step 1, generate the shared question containing "[entity_name]" which can be replaced by the two entity names and then have different answers accordingly.<br>Step 2, use roughly the same amount of words to answer the questions for both entities separately.<br>Step 3, generate a misleading distractor for both entities separately. The distractors should have roughly the same length with the correct answers.<br>Step 4, form the final multiple choice question using the above four answer candidates, and make sure the answer for Entity1 is 'A', the answer for Entity2 is 'B', and their misleading answer candidates are 'C' and 'D'.<br>Step 5, check whether the final question and answer candidates meet the above requirements. If yes, then output **SCUUEED**, otherwise output **FAIL**.<br><br>Follow these examples:<br>…….(Examples written by experts)<br><br>Hypernym: [Hypernym]<br>Entity1: [Entity1]<br>Entity1 Description: [Entity1 Description]<br>Entity2: [Entity2]<br>Entity2 Description: [Entity2 Description]</td>
  </tr>
</table>

Table 7: The prompt used to generate questions in COMPARISONQA. Placeholders [Hypernym], [Entity1], [Entity1 Description], [Entity2], [Entity2 Description] will be replaced with real hypernym, high-frequency entity, low-frequency entity, and their descriptions accordingly.

9,166 different hypernyms.

In the process of Abstract Question Generation, the prompt we use to generate questions is shown in Table 7. In questions generated by LLMs, the answer for the high-frequency entity is \( A \), low-frequency entity is \( B \), and their distractors are \( C \) and \( D \) respectively. And in the end, we randomly shuffle the 4 options in one question to guarantee the balance of the correct options.

<table>
  <tr>
    <th>Task</th>
    <th>Prompt</th>
  </tr>
  <tr>
    <td>Uncertainty Generation</td>
    <td>Question: [question]<br>Answer: **[option]**<br>Uncertainty: **[uncertainty percentage]**<br>Answer the following multiple choice question. Select only one correct answer from the choices and give your uncertainty score, following the above format.<br><br>[Question]<br>A. [OptionA]. B. [OptionB].<br>C. [OptionC]. D. [OptionD].</td>
  </tr>
</table>

Table 8: The prompt used to generate uncertainty score for proprietary LLMs. Placeholders [Hypernym], [Entity1], [Entity1 Description], [Entity2], [Entity2 Description] will be replaced with real hypernym, high-frequency entity, low-frequency entity, and their descriptions accordingly.

D Calculation of Uncertainty

Following Liu et al. (2023), we first combine the questions with each of their options, and use GPT-4o-mini to transform them into four statements. The data can also be found in our benchmark.

Then we compute the uncertainty for each statement. For open-source LLMs, uncertainty refers to their perplexity for generating the correct statement. For proprietary LLMs, we allow them to generate their uncertainty scores, with prompt shown in Table 8. Then, the threshold between high and low uncertainty for each model is determined by its own average uncertainty on the testing set of each benchmark respectively.

There are several reasons for choosing different uncertainties for each setting. On one hand, perplexity-based uncertainty is unavailable for proprietary LLMs. On the other hand, open source LLMs often fail to understand the instructions and can not generate uncertainty scores. We have conducted experiments on verbalized uncertainty of open source LLMs. However, experiments show that Llama-3-8B-Instruct has a 18.5% chance of not generating an uncertainty score, which makes the results unreliable and will also affect further robustness evaluations. In addition, Vashurin et al.
(2024) mentioned that, for multiple choice QA, information-based methods such as perplexity are substantially superior to quantifying model uncertainty.

<table>
  <tr>
    <th>Method</th>
    <th>Prompt</th>
  </tr>
  <tr>
    <td>Zero-Shot</td>
    <td>[Question]<br>A. [OptionA]. B. [OptionB].<br>C. [OptionC]. D. [OptionD].<br>The correct answer is:</td>
  </tr>
  <tr>
    <td>Few-Shot</td>
    <td>[Examples]<br>Answer the multiple choice question. Select only one correct answer from the choices, following above examples.<br><br>[Question]<br>A. [OptionA]. B. [OptionB].<br>C. [OptionC]. D. [OptionD].</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>Question: [question]<br>Rational: [rationale]<br>Answer: **[option]**<br>Answer the multiple choice question. Think step by step and generate a short rationale to support your reasoning. Choose one best answer based on the generated rational, following the above format. Keep your whole response in 50 tokens.<br><br>[Question]<br>A. [OptionA]. B. [OptionB].<br>C. [OptionC]. D. [OptionD].</td>
  </tr>
</table>

Table 9: The prompt used when evaluating LLMs on our benchmark. Placeholders [Examples], [Question], [OptionA], [OptionB], [OptionC], [OptionD] will be replaced with the real examples, questions and their options accordingly.

E Experiment Details

For the main evaluations on COMPARISONQA, we categorize the evaluation of different models into three types: (1) OPEN SOURCE LLM ZERO-SHOT: We first evaluate Llama3, Llama3.1, Llama3.2 (Touvron et al., 2023; Dubey et al., 2024), Gemma2 (Mesnard et al., 2024; Riviere et al., 2024), Phi3.5 (Abdin et al., 2024), Falcon, Falcon2 (Malartic et al., 2024), Mistral (Jiang et al., 2023), and their instruction versions accordingly in a zero-shot manner (Qin et al., 2023). (2) OPEN SOURCE LLM FEW-SHOT: Then we evaluate the above models in a few-shot manner (Brown et al., 2020). Since our benchmark is in the form of four options multiple-choice questions, the shot number is set to four to minimize bias, where each of the four examples corresponds to a different correct answer in (a, b, c, d). (3) PROPRIETARY LLM API: Finally, we evaluate the performance of GPT-4o (OpenAI, 2023, 2024b) and GPT-4o-mini (OpenAI, 2024a), using zero-shot, few-shot, and Chain-of-Thought (CoT; Wei et al., 2022).

All the open-source models are run on 4 NVIDIA A6000 (40G) GPUs with BF32. And for proprietary LLM, we access them via OpenAI API 2. The different kinds of prompts we use are shown in Table 9. And all the evaluation results are reported in Table 10.

For the question filtering of COMPARISONQA-Hard, the six open-source LLMs we use are Llama-3-8B-Instruct, Llama-3.1-8B-Instruct, gemma-2-9b, Phi-3.5-mini-instruct, Falcon-11B, and Mistral-7B-Instruct-v0.3.

2https://platform.openai.com/docs/api-reference
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="3">High Freq Question</th>
    <th colspan="3">Low Freq Question</th>
    <th colspan="3">Average</th>
    <th colspan="3">Difference (H -> T)</th>
  </tr>
  <tr>
    <th>Unc.</th><th>Acc.</th><th>Ma-F1</th>
    <th>Unc.</th><th>Acc.</th><th>Ma-F1</th>
    <th>Unc.</th><th>Acc.</th><th>Ma-F1</th>
    <th>Unc.</th><th>Acc.</th><th>Ma-F1</th>
  </tr>
  <tr>
    <td>Random</td>
    <td>-</td><td>25.29</td><td>25.29</td>
    <td>-</td><td>25.22</td><td>25.22</td>
    <td>-</td><td>25.26</td><td>25.26</td>
    <td>-</td><td>0.07</td><td>0.07</td>
  </tr>
  <tr>
    <td>Majority</td>
    <td>-</td><td>25.70</td><td>10.22</td>
    <td>-</td><td>25.14</td><td>10.04</td>
    <td>-</td><td>25.42</td><td>10.13</td>
    <td>-</td><td>0.56</td><td>0.18</td>
  </tr>
  <tr>
    <th colspan="13">LLM (Open Source) + Zero-shot</th>
  </tr>
  <tr>
    <td>Llama-3 sb</td><td>54.33</td><td>65.90</td><td>63.60</td><td>81.54</td><td>53.83</td><td>51.29</td><td>67.94</td><td>59.87</td><td>57.44</td><td>↑6.11</td><td>↓12.07</td><td>↓12.30</td>
  </tr>
  <tr>
    <td>Llama-3-Instr sb</td><td>77.55</td><td>80.72</td><td>80.71</td><td>117.41</td><td>69.03</td><td>68.95</td><td>97.48</td><td>74.88</td><td>74.83</td><td>↑39.86</td><td>↓11.69</td><td>↓11.76</td>
  </tr>
  <tr>
    <td>Llama-3.1 sb</td><td>55.91</td><td>65.29</td><td>63.31</td><td>83.35</td><td>52.66</td><td>50.52</td><td>69.63</td><td>58.98</td><td>56.92</td><td>↑27.44</td><td>↓12.63</td><td>↓12.78</td>
  </tr>
  <tr>
    <td>Llama-3.1-Instr sb</td><td>58.97</td><td>80.06</td><td>80.08</td><td>87.05</td><td>69.99</td><td>69.94</td><td>73.01</td><td>75.03</td><td>75.01</td><td>↑28.08</td><td>↓10.07</td><td>↓10.14</td>
  </tr>
  <tr>
    <td>Llama-3.2 sb</td><td>67.51</td><td>57.93</td><td>53.65</td><td>99.55</td><td>48.10</td><td>44.66</td><td>83.53</td><td>53.02</td><td>49.16</td><td>↑32.04</td><td>↓9.83</td><td>↓9.00</td>
  </tr>
  <tr>
    <td>Llama-3.2-Instr sb</td><td>74.27</td><td>71.47</td><td>71.51</td><td>108.24</td><td>62.17</td><td>62.18</td><td>91.26</td><td>66.82</td><td>66.85</td><td>↑33.97</td><td>↓9.30</td><td>↓9.33</td>
  </tr>
  <tr>
    <td>Gemma-2 2B</td><td>133.57</td><td>46.48</td><td>43.40</td><td>213.40</td><td>37.76</td><td>34.07</td><td>173.49</td><td>42.12</td><td>38.74</td><td>↑79.83</td><td>↓8.72</td><td>↓9.33</td>
  </tr>
  <tr>
    <td>Gemma-2 9B</td><td>124.97</td><td>64.50</td><td>64.80</td><td>211.34</td><td>52.24</td><td>51.23</td><td>168.16</td><td>58.37</td><td>58.01</td><td>↑86.37</td><td>↓12.26</td><td>↓13.57</td>
  </tr>
  <tr>
    <td>Phi-3.5-mini-Instr 4B</td><td>27.81</td><td>72.81</td><td>72.78</td><td>39.80</td><td>65.26</td><td>65.00</td><td>33.81</td><td>69.04</td><td>68.89</td><td>↑11.99</td><td>↓7.55</td><td>↓7.78</td>
  </tr>
  <tr>
    <td>Falcon 7B</td><td>62.33</td><td>18.19</td><td>18.55</td><td>92.92</td><td>17.93</td><td>18.19</td><td>77.63</td><td>18.06</td><td>18.37</td><td>↑30.59</td><td>↓0.26</td><td>↓0.36</td>
  </tr>
  <tr>
    <td>Falcon-Instr 7B</td><td>88.88</td><td>25.68</td><td>14.49</td><td>128.75</td><td>25.71</td><td>14.01</td><td>108.82</td><td>25.70</td><td>14.25</td><td>↑39.87</td><td>↑0.02</td><td>↑0.48</td>
  </tr>
  <tr>
    <td>Falcon2 11B</td><td>56.93</td><td>70.72</td><td>69.80</td><td>87.31</td><td>58.07</td><td>56.70</td><td>72.12</td><td>64.40</td><td>63.25</td><td>↑30.38</td><td>↓12.65</td><td>↓13.10</td>
  </tr>
  <tr>
    <td>Mistral-v0.3 7B</td><td>39.83</td><td>65.55</td><td>63.11</td><td>56.99</td><td>53.36</td><td>50.26</td><td>48.41</td><td>59.46</td><td>56.69</td><td>↑17.16</td><td>↓12.19</td><td>↓12.85</td>
  </tr>
  <tr>
    <td>Mistral-v0.3-Instr 7B</td><td>44.73</td><td>73.53</td><td>73.14</td><td>66.09</td><td>63.05</td><td>62.40</td><td>53.41</td><td>68.29</td><td>67.77</td><td>↑21.36</td><td>↓10.48</td><td>↓10.74</td>
  </tr>
  <tr>
    <th colspan="13">LLM (Open Source) + 4-shot</th>
  </tr>
  <tr>
    <td>Llama-3 sb</td><td>21.89</td><td>75.57</td><td>75.55</td><td>23.75</td><td>61.00</td><td>61.01</td><td>22.82</td><td>68.29</td><td>68.28</td><td>↑1.86</td><td>↓14.57</td><td>↓14.55</td>
  </tr>
  <tr>
    <td>Llama-3-Instr sb</td><td>26.20</td><td>79.94</td><td>79.92</td><td>28.70</td><td>67.98</td><td>67.95</td><td>27.45</td><td>73.96</td><td>73.93</td><td>↑2.50</td><td>↓11.96</td><td>↓11.96</td>
  </tr>
  <tr>
    <td>Llama-3.1 sb</td><td>20.82</td><td>74.91</td><td>74.89</td><td>22.62</td><td>62.00</td><td>62.00</td><td>21.72</td><td>68.46</td><td>68.45</td><td>↑1.81</td><td>↓12.91</td><td>↓12.90</td>
  </tr>
  <tr>
    <td>Llama-3.1-Instr sb</td><td>20.63</td><td>79.74</td><td>79.74</td><td>22.48</td><td>69.09</td><td>69.07</td><td>21.56</td><td>74.42</td><td>74.40</td><td>↑1.85</td><td>↓10.65</td><td>↓10.66</td>
  </tr>
  <tr>
    <td>Llama-3.2 sb</td><td>23.58</td><td>68.89</td><td>68.84</td><td>25.59</td><td>57.26</td><td>57.17</td><td>24.59</td><td>63.08</td><td>63.01</td><td>↑2.01</td><td>↓11.63</td><td>↓11.67</td>
  </tr>
  <tr>
    <td>Llama-3.2-Instr sb</td><td>23.66</td><td>71.43</td><td>71.43</td><td>25.67</td><td>62.12</td><td>62.09</td><td>24.67</td><td>66.78</td><td>66.76</td><td>↑2.01</td><td>↓9.32</td><td>↓9.34</td>
  </tr>
  <tr>
    <td>Gemma-2 2B</td><td>26.96</td><td>62.99</td><td>62.93</td><td>29.43</td><td>50.93</td><td>50.91</td><td>28.20</td><td>56.96</td><td>56.92</td><td>↑2.47</td><td>↓12.06</td><td>↓12.02</td>
  </tr>
  <tr>
    <td>Gemma-2 9B</td><td>20.26</td><td>80.10</td><td>80.08</td><td>22.36</td><td>68.36</td><td>68.31</td><td>21.31</td><td>74.23</td><td>74.20</td><td>↑2.10</td><td>↓11.74</td><td>↓11.77</td>
  </tr>
  <tr>
    <td>Phi-3.5-mini-Instr 4B</td><td>11.02</td><td>73.68</td><td>73.67</td><td>11.85</td><td>67.46</td><td>67.33</td><td>11.44</td><td>70.57</td><td>70.50</td><td>↑0.83</td><td>↓6.22</td><td>↓6.34</td>
  </tr>
  <tr>
    <td>Falcon 7B</td><td>21.69</td><td>28.40</td><td>21.25</td><td>23.42</td><td>28.25</td><td>20.63</td><td>22.56</td><td>28.33</td><td>20.94</td><td>↑1.73</td><td>↓0.15</td><td>↓0.62</td>
  </tr>
  <tr>
    <td>Falcon-Instr 7B</td><td>21.69</td><td>26.27</td><td>18.32</td><td>23.42</td><td>25.67</td><td>18.15</td><td>22.56</td><td>25.97</td><td>18.23</td><td>↑1.73</td><td>↓0.59</td><td>↓0.17</td>
  </tr>
  <tr>
    <td>Falcon2 11B</td><td>16.42</td><td>77.11</td><td>77.01</td><td>17.77</td><td>65.92</td><td>65.75</td><td>17.10</td><td>71.52</td><td>71.38</td><td>↑1.35</td><td>↓11.19</td><td>↓11.26</td>
  </tr>
  <tr>
    <td>Mistral-v0.3 7B</td><td>13.89</td><td>75.55</td><td>75.53</td><td>14.99</td><td>62.88</td><td>62.85</td><td>14.44</td><td>69.22</td><td>69.19</td><td>↑1.10</td><td>↓12.67</td><td>↓12.68</td>
  </tr>
  <tr>
    <td>Mistral-v0.3-Instr 7B</td><td>15.97</td><td>74.46</td><td>74.40</td><td>17.40</td><td>65.51</td><td>65.35</td><td>16.69</td><td>69.99</td><td>69.87</td><td>↑1.43</td><td>↓8.95</td><td>↓9.05</td>
  </tr>
  <tr>
    <th colspan="13">LLM (Proprietary) API</th>
  </tr>
  <tr>
    <td>GPT4o-mini (Zero-Shot)</td><td>13.52</td><td>85.61</td><td>85.58</td><td>18.34</td><td>73.85</td><td>73.73</td><td>15.93</td><td>79.73</td><td>79.66</td><td>↑4.82</td><td>↓11.76</td><td>↓11.85</td>
  </tr>
  <tr>
    <td>GPT4o-mini (Few-Shot)</td><td>25.74</td><td>84.78</td><td>84.69</td><td>38.17</td><td>72.76</td><td>72.47</td><td>31.96</td><td>78.77</td><td>78.58</td><td>↑12.43</td><td>↓12.02</td><td>↓12.22</td>
  </tr>
  <tr>
    <td>GPT4o-mini (CoT)</td><td>10.53</td><td>86.25</td><td>86.25</td><td>12.27</td><td>74.39</td><td>74.40</td><td>11.40</td><td>80.32</td><td>80.32</td><td>↑1.74</td><td>↓11.85</td><td>↓11.85</td>
  </tr>
  <tr>
    <td>GPT4o (Zero-Shot)</td><td>14.18</td><td>93.86</td><td>93.95</td><td>30.98</td><td>85.76</td><td>86.69</td><td>22.58</td><td>89.81</td><td>90.32</td><td>↑16.80</td><td>↓8.10</td><td>↓7.26</td>
  </tr>
  <tr>
    <td>GPT4o (Few-Shot)</td><td>28.41</td><td>93.94</td><td>93.95</td><td>45.81</td><td>86.54</td><td>86.75</td><td>37.11</td><td>90.24</td><td>90.35</td><td>↑17.40</td><td>↓7.40</td><td>↓7.20</td>
  </tr>
  <tr>
    <td>GPT4o (CoT)</td><td>10.39</td><td>92.40</td><td>92.47</td><td>18.36</td><td>85.47</td><td>85.72</td><td>14.38</td><td>88.93</td><td>89.10</td><td>↑7.97</td><td>↓6.93</td><td>↓6.75</td>
  </tr>
</table>

Table 10: Performance of various LLMs on the testing set of COMPARISONQA. Unc., Acc., and Ma-F1, denote Uncertainty, Accuracy, and Macro F1-score. And the Difference column shows how scores change from high-frequency questions to low-frequency questions. The best performances within each method are underlined and the best among all methods are bold-faced. And for the Difference column, We underline the largest difference within each method and bold the one among all methods.

Reviewer paper summaries: The paper presents ComparisonQA, a dataset consisting of 283K instances designed to evaluate the ability of LLMs to answer questions about both low-frequency and high-frequency entities under identical conditions. It introduces an automated procedure for dataset construction, a two-step evaluation methodology, and assesses 16 models on the proposed task. The findings highlight significant underperformance by most models on low-frequency entity questions.

This paper introduces a new dataset, ComparisonQA, where each sample consists of a pair of multiple-choice questions that share the same abstract question but differ in entities (high-frequency vs. low-frequency entities).
The dataset is used to evaluate the factual knowledge of LLMs.
To avoid shortcuts, the authors propose a two-round evaluation method based on two metrics: correctness and uncertainty, to assess the robustness of factual knowledge in LLMs. Additionally, they introduce an enhanced version of the dataset, ComparisonQA-hard, which filters out low-quality questions and those prone to shortcuts.
Experimental results show that LLMs achieve high performance—and also robustness—on high-frequency knowledge but lower performance on low-frequency knowledge.

This paper introduces ComparisonQA,  an evaluation benchmark to measure the factual robustness of LLMs by controlling for both knowledge frequency and entity frequency. They also propose a two-round evaluation framework for tricky samples. They observe the low robustness of LLMs toward low-frequency knowledge.
Summary of strengths: Strengths:
1. The studied problem addresses a critical area of QA, highlighting an aspect that requires further attention and exploration.
2. The collected dataset serves as a valuable resource for testing and fine-tuning models, with the construction procedure being a noteworthy contribution.
3. The critique of multiple-choice evaluation is well-founded and opens a meaningful discussion on assessment methods.
4. The comprehensive set of evaluated models is impressive and provides valuable insights for readers seeking to select a QA model.

- It’s interesting to have pairs of questions featuring high- and low-frequency entities.

- The dataset can be useful to the QA research community.

1. This paper introduces ComparisonQA, a novel benchmark addressing the limitations of entity-centric evaluations.
2. This paper proposes a two-round evaluation, a simple yet effective framework using correctness and uncertainty.
3. This paper conducts extensive experiments with numerous models and diverse setups.
Summary of weaknesses: ## Major Weakness:

1. The benchmark is not on comparative questions or comparisons, which might mislead the readers. There exists a specific branch of QA devoted to comparative questions and is a mixture of argument mining, question answering, and RAG.
2. Lack of experimental details. It raises numerous questions and makes it difficult to fully understand the proposed procedures and methodologies (see comments for questions)
3. The paper provides no details regarding the labeling scheme, instructions given to annotators, or how agreement among them was measured. This lack of transparency raises concerns about the reliability of the human assessment process.
4. The reported 95% correctness rate implies 5% incorrect labels, equating to approximately 14,000 incorrect instances in the dataset. Given the dataset's split, there is a possibility that these incorrect labels could disproportionately affect the test set, potentially constituting a significant portion (up to half) of it.
5. It is highly problematic to calculate uncertainty using different procedures without explicitly mentioning this in the main tables, as it undermines the validity of comparisons. Prior research demonstrates significant performance differences between verbalized uncertainty and perplexity-based uncertainty measures [1].
6. The paper lacks justification for the choice of uncertainty estimation methods. There are more advanced and reliable approaches for both black-box and white-box uncertainty estimation [1], which should have been considered or at least discussed.
Additionally, the omission of chain-of-thought prompting for the instruct versions of open-source models is unexplained. 
7. The finding that low-frequency questions are more challenging to solve is neither novel nor particularly interesting, as it aligns with prior research [2]. Similarly, the observation of lower uncertainty for high-frequency entities, particularly when measured by perplexity, is unsurprising. This directly reflects the training process, where the loss is naturally lower for more frequent instances.
8. line 330: the takeaway is not consistent with the data and might be biased. The data suggests that chain-of-thought prompting improves performance for GPT-4o-mini, as evidenced in the provided table. Moreover, GPT-4o performs CoT reasoning by default in zero-shot scenarios. Without examples of the model outputs, it is difficult to validate the result and its interpretation.
9. The first step of the procedure described in Section 4.1 focuses on evaluating LLM overconfidence and underconfidence, a topic that has been previously studied [3]. However, this approach is not well-suited to assess whether the LLM possesses robust knowledge. 
- Uncertainty measures, particularly in the case of verbalized uncertainty, are often poorly aligned with the model's actual knowledge, making this evaluation unreliable.
- While the second step of the procedure is more intriguing, its implementation raises concerns. Limiting the evaluation to questions related to uncertainty and treating them as indicative of knowledge seems arbitrary. A more comprehensive approach would involve applying this procedure to all questions in the dataset.
- Additionally, the method of breaking a question into four sub-questions and requiring the model to correctly answer all four introduces an unequal setting. This approach may unfairly penalize the model, as it amplifies the impact of any single error on the overall result.
10. The creation of the "hard benchmark" appears to rely on selecting incorrectly answered questions from the main dataset. While this approach may highlight challenging instances, it offers limited novelty compared to the main dataset itself. Additionally, this procedure could be applied to any dataset to create a "hard" subset, making it less unique as a contribution. While the intention to identify difficult examples is understandable, the approach may not provide significant new insights or advance the understanding of model performance.


## Minor Weakness:
* The example provided in the introduction does not align with the focus of the study. Specifically, the question about Einstein's favorite fruit does not involve low-frequency entities, which contradicts the core premise of evaluating model performance on low- versus high-frequency entities. This inconsistency may confuse readers and detract from the relevance of the example to the study's objectives.

[1] Vashurin, Roman, et al. "Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph." arXiv preprint arXiv:2406.15627 (2024).
[2] Mallen, Alex, et al. "When not to trust language models: Investigating effectiveness of parametric and non-parametric memories." arXiv preprint arXiv:2212.10511 (2022).
[3] Ni, Shiyu, et al. "When Do LLMs Need Retrieval Augmentation? Mitigating LLMs' Overconfidence Helps Retrieval Augmentation." arXiv preprint arXiv:2402.11457 (2024).

1. There is a lack of human performance evaluation on the dataset.

2. The method for calculating the uncertainty metric differs between open-source and proprietary LLMs (Appendix D). I’m not sure whether this difference affects the results in Table 1 (e.g., the uncertainty in zero-shot for open-source LLMs is higher than in few-shot, whereas the opposite is true for proprietary LLMs).

3. All of their questions are generated using LLMs. Although the authors provide “expert verification scores” for a sample of 200 question pairs (Section 3.3), we cannot ensure the quality of the entire dataset.

1. ComparisonQA dataset includes one high-frequency entity and low-frequency entity per sample. In other words, the evaluation results using ComparisonQA may overly rely on certain entities, not generalizing LLMs’ knowledge toward overall entities relevant to the hypernym. Adding more entities to the same questions/hypernyms and examining whether high/low-frequency entities show similar trends with respect to their entity frequency will make this paper more convincing.
Comments, suggestions and typos: Questions: 
1. The rationale behind the sampling procedure for low- and high-frequency entities is unclear. Why are 1,000 random samples taken, and how does accounting for the percentage of relations contribute to this method? 
2. The necessity for questions about two entities to yield different answers is not well-justified. Why can’t both entities have the same answer?
3. Prompts used for the dataset construction procedure are not provided, leaving an essential aspect of the methodology unexplained.
4. It is unclear whether the uncertainty threshold was determined using the training set, and if so, how this was done.
5. The explanation of few-shot examples is ambiguous. Why are there specifically four few-shots, corresponding to the four options? This approach lacks clarity and could equally apply to other configurations, such as a 15-shot setup with a question and four options. And more importantly, what are the few-shots?
6. The generation of verbalized perplexity is not well-documented. What prompts were used to generate it, and how was the process implemented?

- #028: there should be no space before the footnote.

- Table 1: the increase in the uncertainty metric for Gemma-2 is 86.37 in zero-shot and 2.10 in few-shot. Is this gap too large? Do you have any explanations for this? Also, for GPT-40-mini, the increase in the uncertainty score in few-shot is higher than in zero-shot, which is the opposite of the trend observed in open-source LLMs.

- I would like to take a look at the data and code but I can’t find the supplementary materials.

- The paper mentions the uncertainty metric extensively, but its calculation is only described in the Appendix. Since your findings also discuss this metric, I think that the calculation should be included in the main text. (The same applies to the definition of Knowledge Frequency.)

- #278: “a total of 283,455 abstract questions” how do you obtain these abstract questions? In line #1136-1138, you mention that, “293K entities pairs from 9,261 hypernyms in total. These pairs are then fed into the LLM to generate questions” Could you provide more details about the process of obtaining the 283,455 abstract questions?

1. According to Appendix D, this paper adopts two different uncertainty measures for open and proprietary models due to the inaccessible internal stats of proprietary models. Providing a gap between the two measures in open models (i.e., how big the difference is between perplexity-based scores and self-judged scores in a single open model) would help readers understand the gap between the two measures.
Overall assessment scores: 3.5

3

3.5
Metareview score: 4
Metareview: The paper introduces a benchmark for evaluating QA performance for LLMs on question pairs which differ only in the frequency of a question entity.
Summary of reasons to publish: This is a clever idea which allows clearer understanding of the role of entity frequency in QA performance and may help in understanding fundamental issues like parameter/context dependencies and hallucination.