RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
A.7 Full Consistency Analysis
In this section, we describe the consistency analysis for all values of our truth table \mathcal{A} , introduced in section 3.2 .
Apart from the four categories described before, the truth table outcome can also be “Ambiguous Question” if \mathcal{A}=(\texttt{y},\texttt{y},\texttt{n}) , as both steps succeeded but converged to different answers (meaning the question had more than one possible correct answer).
Another option is for the mistakes to cancel out, which is a rare scenario \mathcal{A}=(\texttt{n},\texttt{n},\texttt{y}) where the model generated an inaccurate question and answered its own question incorrectly, but managed to arrive at the original entity a . The final category is a Metric Prediction Error, a scenario that only occurs if either just QA or RQA was predicted to fail, but a and \hat{a} were predicted to be matching ( \mathcal{A}=(\texttt{n},\texttt{y},\texttt{y}) or \mathcal{A}=(\texttt{y},\texttt{n},\texttt{y}) ). These scenarios are summarized in Table 4 .

In Figure 8 , we report the full consistency analysis for all 16 of our LLMs and all truth table scenarios.
First, we note that the four categories reported in Figure 3 encompass the majority of the truth table.
Second, even for smaller LLMs, our claims hold; LLMs can often detect their own question inaccuracies from RQA through QA .
A.8 Further Analysis of RQA Questions
Due to page limit constraints of a short paper, we were unable to show the entire qualitative analysis we conducted on questions generated in RQA .
Below, we give more qualitative results on the answerability of questions from RQA (Appendix A.8.1 ), a cross-model comparison of question duplicates in RQA (Appendix A.8.2 ), the ability of LLMs to match the ground-truth question during RQA (Appendix A.8.3 ), and a brief investigation into memorization in the RQA task (Appendix A.8.4 ).
A.8.1 Are RQA questions unanswerable?
We now seek to understand the types of RQA questions generated in the Number+Text setting, complementing our analysis in section 3.4 .
The Number+Text questions have higher variance and cannot be as neatly categorized as in section 3.4 (e.g. single-step computation). So instead, we study the answerability of 30 generated questions from each LLM, i.e., if the question is clear but leads to an incorrect answer, or if there is an issue with the question that makes it difficult to answer.
We adopt five categories of unanswerable questions from Rogers et al. ( 2023 ) : 1) Invalid Premise: the question contains a false assumption, so it is impossible to answer. For example, Opus generates the question How old was the world’s oldest tortoise, Jonathan, when he passed away in 2022? , but this Tortoise is still alive. 2) No Consensus on the Answer: there is not a single, agreed-upon answer to the question. For example, LLaMA generates the question What is the unique property of the Lie algebra E8 that makes it particularly interesting in theoretical physics? , but there are many unique, interesting properties of Lie algebra that would answer the question. 3) Information not yet Discovered: the answer to this question is currently unknown. For example, GPT-4 generates the question How long, in terms of word count, is the sentence that holds the record for being the longest in the English language without using any punctuation? , but it is currently unknown what could theoretically be the longest sentence. 4) Missing Information: There is not enough information in the question, or it is too vague. For example, GPT-4 generates the question How many individuals attended the annual community festival last year according to the final headcount? , which cannot be answered without knowing more details. 5) Answerable: The question has one right answer.

As expected, when RQA succeeds, questions are mostly answerable (Figure 9 ).
However, we find that a non-trivial proportion of generated questions when RQA fails are unanswerable, reaching nearly 60% for GPT-4.
The most common types of unanswerable questions are those that are missing information, meaning that they are too vague or ambiguous, or those that have false premises or assumptions.
While several works explore methods to answer ambiguous questions Min et al. ( 2020 ); Kim et al. ( 2023a ) or questions with false presuppositions Yu et al. ( 2023 ); Kim et al. ( 2023b ) , our analysis reveals a need to avoid generating ambiguous or faulty-presupposition questions in RQA .

In Tables 5 and 6 , we provide examples of question/error types in our qualitative analysis on the Number and Number+Text split, respectively.
A.8.2 Do LLMs give the same RQA questions?
While most of our analysis treated LLMs independently, we now study whether LLMs generate the same exact questions (i.e. duplicates) in RQA .
Figure 11 shows that LLMs more frequently generate duplicated questions across entities versus matching questions from other models. For example, LLaMA-3 70B generates 379 duplicate questions in the Numbers setting, even when the input answer is altered. This aligns with very recent work suggesting that LLMs may often conduct pattern-matching rather than engaging in true, generalizable reasoning Mirzadeh et al. ( 2024 ) .

We also note that interestingly, models in the same family will more likely generate duplicated questions. For example, GPT-3.5, GPT-4, and GPT-4o generate the same questions in RQA more often than when compared to other LLM families.
Thus, we speculate that these model families likely share similar pre-training and alignment data, which is optimized on through different training recipes.
A.8.3 Does RQA match the gold question?
We now explore whether the questions generated for an answer in RQA match the gold question we collected for that answer.
When determining if the two questions are semantically equivalent, we follow the protocol of Balepur et al. ( 2024b ) and analyze whether the two questions test the exact same knowledge.
Figure 10 shows that the LLMs can often match the true question when RQA succeeds in Number+Text settings, reaching as high as 40% of cases for GPT-4; the questions never matched for Number.
One explanation for the high match rate is dataset contamination Ishihara ( 2023 ) , but it is also possible that the most likely question the LLM abductively reason towards is the ground-truth question.
For example, for the answer “120 counties,” the only salient fact linked to the entity is that Kentucky has 120 counties McCarthy ( 1959 ) ; this led GPT-4’s question and the ground-truth question to both ask about Kentucky.
A.8.4 Are any RQA questions memorized?
Since the presence of duplicates in Appendix A.8.3 suggests that LLMs may just be retrieving similar questions from pretraining rather than reasoning towards new questions in RQA , we now investigate the novelty of the RQA questions Merrill et al. ( 2024 ) , i.e., whether they are exactly copied from pretraining.
We do not know which corpora all of our LLMs are trained on, so we use the Dolma Soldaini et al. ( 2024 ) corpus as a proxy for pretraining data.
For each generated RQA question \hat{q} , we compute how frequently the exact question \hat{q} appears in Dolma via infini-gram Liu et al. ( 2024b ) .

Table 7 reveals that in total, 2.87 % of RQA generated questions are exactly found in Dolma. For comparison, 1.25 % of our ground-truth questions are present in Dolma.
While we did not explicitly instruct the model to generate a new question that it has not seen in pretraining, practitioners may need to design specialized techniques if they desire more novel questions from RQA .

When comparing exact question match frequency by model, weaker/smaller LLMs tend to copy more from pretraining data, suggesting that smaller LLMs are more prone to RQA memorization.
Further, the Hard Fact setting is much less prone to question copying in RQA , likely because the RQA input answers have very low pretraining token count ( section 3.3 ), which also further confirms that LLMs may struggle to retrieve exact pretraining knowledge for long-tail facts Kandpal et al. ( 2023 ) .

We present examples of RQA questions that appear the most in Dolma in Table 8 .
The tendency to generate inaccurate or ambiguous questions may be influenced by pretraining, as many of these questions appear directly in Dolma.
Figure 1: GPT-4 consistency check in RQA / QA . The LLM fails to produce a valid question with answer 488 (top), but correctly gives the answer 77 for its own question (bottom).; GPT-4 consistency check in RQA / QA . The LLM fails to produce a valid question with answer 488 (top), but correctly gives the answer 77 for its own question (bottom).
Figure 2: LLM RQA ( blue ) and QA ( red ) accuracy with 95% CIs for metric error rate. LLMs are much weaker in abductive RQA in numerical settings (Number/Number+Text), but in text settings (Easy/Hard Entity), deductive QA is slightly weaker.; LLM RQA ( blue ) and QA ( red ) accuracy with 95% CIs for metric error rate. LLMs are much weaker in abductive RQA in numerical settings (Number/Number+Text), but in text settings (Easy/Hard Entity), deductive QA is slightly weaker.
Figure 3: Logical consistency of RQA and QA . For Number and Number+Text entities, most LLMs lack consistency (except GPT-4o), with RQA as the main failure point. Otherwise, LLMs are fairly consistent, with QA as the failure point for Hard Factual Entities. We display the strongest LLM from each model family for brevity, with all results shown in Appendix A.7 .; Logical consistency of RQA and QA . For Number and Number+Text entities, most LLMs lack consistency (except GPT-4o), with RQA as the main failure point. Otherwise, LLMs are fairly consistent, with QA as the failure point for Hard Factual Entities. We display the strongest LLM from each model family for brevity, with all results shown in Appendix A.7 .
Figure 4: Answer token count and question difficulty of when RQA succeeds/fails, averaged over all LLMs.; Answer token count and question difficulty of when RQA succeeds/fails, averaged over all LLMs.
Figure 5: Analysis of Number RQA errors. RQA often fails when the LLM tries to give a multi-step question.; Analysis of Number RQA errors. RQA often fails when the LLM tries to give a multi-step question.
Figure 6: LLM deduction ( blue ) and abduction ( red ) accuracy based on human annotations on a subset of data (20 labels per model/dataset). The plot shows a similar trend as the automated metrics (LLMs are weaker in abduction in numerical settings, but stronger in abduction in non-numerical settings), confirming the validity of our metrics.; LLM deduction ( blue ) and abduction ( red ) accuracy based on human annotations on a subset of data (20 labels per model/dataset). The plot shows a similar trend as the automated metrics (LLMs are weaker in abduction in numerical settings, but stronger in abduction in non-numerical settings), confirming the validity of our metrics.
Figure 7: LLM deduction ( blue ) and abduction ( red ) accuracy with GPT-4 on numerical entities. For QA , we present the zero-shot prompt used in section 3.1 . For RQA , we test adding chain-of-thought instructions (GPT-4 + CoT), asking the LLM to verify its question post-generation (GPT-4 + Self-Verification), and including five exemplars (GPT-4 + 5-Shot). None of these strategies allow the model to fully match the QA accuracy.; LLM deduction ( blue ) and abduction ( red ) accuracy with GPT-4 on numerical entities. For QA , we present the zero-shot prompt used in section 3.1 . For RQA , we test adding chain-of-thought instructions (GPT-4 + CoT), asking the LLM to verify its question post-generation (GPT-4 + Self-Verification), and including five exemplars (GPT-4 + 5-Shot). None of these strategies allow the model to fully match the QA accuracy.
Figure 8: QA and RQA logical consistency across all models. The consistency trends are also prevalent for smaller/less capable LLMs; RQA and QA consistency is higher for easy/hard entities, but LLMs can often detect their own RQA inaccuracies in numerical settings.; QA and RQA logical consistency across all models. The consistency trends are also prevalent for smaller/less capable LLMs; RQA and QA consistency is higher for easy/hard entities, but LLMs can often detect their own RQA inaccuracies in numerical settings.
Figure 9: Error analysis of questions from RQA on Number+Text. When RQA fails, questions are often unanswerable (30-60%), and frequently include false premises or omit key information that is needed to answer the question.; Error analysis of questions from RQA on Number+Text. When RQA fails, questions are often unanswerable (30-60%), and frequently include false premises or omit key information that is needed to answer the question.
Figure 10: Proportion of RQA questions on Numbers+Text that semantically match the ground-truth question when RQA succeeds. LLaMA-3 70B, GPT-4, and Opus can all match the ground-truth question over 25% of the time.; Proportion of RQA questions on Numbers+Text that semantically match the ground-truth question when RQA succeeds. LLaMA-3 70B, GPT-4, and Opus can all match the ground-truth question over 25% of the time.
Figure 11: Cross-model frequency of questions from RQA that are exact duplicates. LLMs often generate the same question in RQA even though the input answer changes, reaching as high as 379 for LaMA-3 70B on Numbers.; Cross-model frequency of questions from RQA that are exact duplicates. LLMs often generate the same question in RQA even though the input answer changes, reaching as high as 379 for LaMA-3 70B on Numbers.
Description of our collected dataset for question answering and reverse questioning answering tasks.
Dataset details of each split (Number, Number+Text, Easy Entity, Hard Entity), including the number of data instances, average length of answers (in tokens), and average length of questions (in tokens). Tokens are computed using tiktoken .
Raw agreement with human annotators (i.e. accuracy) of seven tested answer equivalence metrics. The best metric for each dataset split is in bold .
All truth table outcomes for the consistency analysis in section 3.2 .
Examples of RQA question types and errors on the Number split.
Examples of RQA question types and errors on the Number+Text split.
Number of generated RQA questions that are exact matches to a question in the Dolma pretraining corpus. On average, models are prone to copying questions from pretraining \sim 3 % of the time. Smaller/weaker LLMs are more susceptible to copying questions from pretraining in RQA . Further, easy facts and numerical answers are more likely to lead to copied questions in RQA versus our hard facts.
Questions generated from RQA that are most frequently found in the Dolma corpus. The LLM’s tendency to generate inaccurate questions (e.g. How many days are there in a year? for 800 ) or ambiguous questions ( What is the answer to this question? ) could be influenced by how often these questions appear in pretraining.