Venue: NAACL 2025

# Reverse Question Answering:

Can an LLM Write a Question so Hard (or Bad) that it Can't Answer?

 Nishant Balepur\({}^{1}\) Feng Gu\({}^{1}\) Abhilasha Ravichander\({}^{2}\) Shi Feng\({}^{3}\)

Jordan Boyd-Graber\({}^{1}\) Rachel Rudinger\({}^{1}\)

\({}^{1}\)University of Maryland \({}^{2}\)University of Washington

\({}^{3}\)George Washington University

{nbaleppur, rudinger}@umd.edu jbg@.umiacs.umd.edu

###### Abstract

Question answering (QA)--giving correct answers to questions--is a popular task, but we test _reverse question answering_ (RQA): for an input answer, give a question with that answer. Past work tests QA and RQA separately, but we test them jointly, comparing their difficulty, adding benchmark design, and checking reasoning consistency. We run 16 llms on QA and RQA with trivia questions/answers, revealing: 1) Versus QA, llMs are much less accurate in RQA for numerical answers, but slightly more accurate in RQA for textual answers; 2) llMs often answer their own invalid questions from RQA accurately in QA, so RQA errors are not from knowledge gaps alone; 3) RQA errors correlate with question difficulty and inversely correlate with answer frequencies in the Dolma corpus; and 4) llMs struggle to provide valid multi-hop questions. By finding question and answer types that lead to RQA errors, we suggest improvements for llm reasoning.1

Footnote 1: Code and data available at [https://github.com/nbalepur/Reverse-QA](https://github.com/nbalepur/Reverse-QA)

## 1 Reversing the Question Answering Task

Question answering (QA) is a long-standing task in nlp [7]. For an input question \(q\), QA deduces the correct answer \(a\)[8]. More recently, large language models (llMs) do the reverse--given an answer \(a\), generate a valid question \(q\) to which \(a\) is the answer--which we call _reverse_**question answering** (RQA).2 RQA thus can be a part of downstream tasks like exam question generation [9] or search query reformulation [5].

Footnote 2: This definition differs from question generation [9], which grounds the answer to an input context.

QA and RQA are often tested separately, but we test them jointly, offering two key benefits. First, it gives insights into open questions on llm abilities, as some show llMs excel in generation over comprehension [10, 11], while others claim verification is easier [11]. Uncovering which task is harder can guide benchmark design [7] and inform data collection practices in writing question-answer pairs (SS3.1; e.g., if RQA is easy, get answers manually and then generate synthetic questions).

Second, chaining RQA and QA forms a consistency check for llm reasoning [13]. RQA--inferring just one of many valid questions--is _abductive_[1], while QA--in inferring an answer from question premises--is _deductive_[10]. Thus, by seeing if QA(RQA\((a))\approx a\), i.e., checking if an llm can answer its own question from RQA (Fig 1), we can assess llMs' logical robustness in abduction and deduction (SS3.2). This analysis can also help determine if llMs can reliably self-verify [12] in downstream RQA tasks like writing exams [10].

To reap these benefits, we test if 16 lmns can produce 1) questions correctly answered by input entities (RQA); and 2) accurate answers for input questions (QA). We collect 3443 trivia question/answer pairs [11], grouped by answer as either numerical or textual entities, forming inputs to evaluate RQA and QA in varied domains.

In numerical domains, llMs are much less accurate in RQA than QA, especially integers (Fig 1); the accuracy difference when llMs do these tasks **exceeded 0.80** for Command-R and LLMaMA-3 (SS3.1).

Figure 1: RQA/QA consistency check using GPT-4. The LLM fails to give a valid question with answer qss (top), but correctly gives the answer qsf for the same question (bottom).



Interestingly, in textual domains, the trend reverses, so llms are not consistently better generators or validators (Li et al., 2024). We then design a consistency check (SS3.2) to see if llms can answer their own RQA questions; numerical RQA failures are not solely due to knowledge gaps, as llMs often **answer their own invalid questions correctly** in QA (33% of cases for Claude-Opus). We then study questions from RQA (SS3.3, SS3.4) and find errors occur when llms give overly-complex, multi-step questions, giving insights into strategies--like complexity bias mitigation in preference data and calibrating models using difficulty scores--to improve lmh RQA reliability. Our contributions are:

1. We use Reverse Question Answering (RQA) to test if llms can provide accurate questions for input answers using abductive reasoning.
2. We reveal many llms have a surprising weakness in RQA on numerical entities, struggling on input answers with lower pretraining token counts and when creating multi-hop questions.
3. We design a consistency check between RQA and QA, showing llms answer their own invalid questions from RQA correctly via QA.

## 2 Experimental Setup

We evaluate llm abilities in question answering (\({\bf Q}\)A) and _reverse_ question answering (\({\bf RQA}\)):

* \({\bf Q}\mathbf{A}(q)\rightarrow\hat{a}\): Given a question \(q\) with a single answer \(a\), the llm produces an answer \(\hat{a}\) for \(q\). QA succeeds if \(a\) matches \(\hat{a}\) semantically. For example, given the input "What is the name of the polygon with three sides?" for \(q\), an llm using QA should give an \(\hat{a}\) that matches "triangle" for \(a\). This typical QA setup tests _deduction_, since the model must reason to the correct answer of \(a\) based on the premises in \(q\).
* \({\bf RQ}\mathbf{A}(a)\rightarrow\hat{q}\): Given an input answer \(a\), the llm must produce a question \(\hat{q}\). RQA succeeds if the correct answer to \(\hat{q}\) is \(a\) (verified via oracle, SS2.3). For example, given the input "triangle" for \(a\), an llm using RQA could succeed with \(\hat{q}\) as "In eight-ball pool, what shape is used to rack the balls?". RQA tests _abduction_, as the model must reason toward one of the many valid questions with the answer \(a\).

This section describes the datasets (SS2.1), models (SS2.2), and metrics (SS2.3) used for RQA and QA.

### Dataset Collection

We study question/answer pairs \((q,a)\) in four domains for QA and RQA inputs, based on \(a\)'s answer type (Table 1). We group them as numerical (Number, Number+Text) or textual (Easy Fact, Hard Fact), providing varied domains for testing.

When \(a\) is a Number, \(q\) is a random, one-step math operation (what is 118+211?). Other types are from qanta[Rodriguez et al., 2019], an expert-curated dataset of multi-sentence trivia qa pairs. For Easy and Hard Facts, \(a\) is the answer to sampled quanta questions, with the last sentence3 as \(q\). We use middle school questions for Easy Facts and college questions for Hard Facts. We obtain Number+Text answers \(a\) in qanta by finding numbers in full questions via regex and \(q\) from the sentence \(a\) appears in. One PhD student checks all QA pairs to ensure they are accurate (details in Appendix A.1).

Footnote 3: qanta questions are paragraph-long and describe a single answer. Sentences in paragraphs are ordered in decreasing difficulty, so we use the last one, forming the easiest question.

### Models

We evaluate 16 llms: GPT (Achiam et al., 2023, 3.5, 4, 4o), Command R (Cohere, 2023, Command-R, Command-R+), Claude (Anthropic, 2023, Sonnet, Haiku, Opus), LLMaM-3 Instruct (Dubey et al., 2024, 8B, 7OB), Yi-1.5 Chat (Young et al., 2024, 6B, 9B, 34B), and Mistral Instruct (Jiang et al., 2024, 7B, 8x7B, 8x22B). All llms use temperature 0. We list all parameters in Appendix A.3.

The QA and RQA prompts are zero-shot, since few-shot exemplars test inductive reasoning, not deduction/abductio

\begin{table}
\begin{tabular}{p{113.8pt} p{113.8pt} p{113.8pt} p{113.8pt}} \hline \hline
**Answer Type** & **Description** & **Example Question** & **Example Answer** & **Count** \\ \hline (1) Number & Integers in \([100,1000)\) & What is 26 times 4? & 104 & 900 \\ (2) Number+Text & Integers with a text entity & When did Pope Hormisdas die? & 523 AD & 743 \\ (3) Easy Fact & Well-known factual entity & Who is the artist that painted Starry Night? & Vincent van Gogh & 900 \\ (4) Hard Fact & Obscure factual entity & What is the final painting by Paolo Uccello? & The Hunt in the Forest & 900 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Description of our collected dataset for Question Answering and Reverse Questioning Answering tasks.

 

Exemplars also do not improve llm RQA accuracy (Appendix A.6). Prompts follow the same template as Figure 1 with format rules to parse outputs Liu et al. (2024c). Two nlp graduate students write the prompts, with all design steps in Appendix A.2.

### Evaluation Metrics

To compute QA accuracy, two graduate students annotate if 1280 llm QA answers \(\hat{a}\) for a question \(q\) match its true answer \(a\) (20 per answer type/model).4 We test seven metrics Li et al. (2024b) that evaluate if \(\hat{a}\) and \(a\) are equivalent. We select DSPy-optimized Khattab et al. (2024) GPT-4o for easy/hard entities and a rule-based method for numerical entities, since these methods had the highest agreement with humans (94% on average).

Footnote 4: The 1280 total annotations are derived from 16 LLMs, 4 splits, and 20 annotations on each llm/split combination.

For RQA accuracy, students annotate if the answer to 1280 questions \(\hat{q}\) from RQA is \(a\) (20 per answer type/model), following rules from Li et al. (2024b). We use DSPy-optimized GPT-4o as an oracle \((\textsc{Verify}^{*}(\hat{q},a))\) to assess if \(a\) answers \(\hat{q}\), which has high (\(90\%\)) human agreement. Metric agreement is high but imperfect, so we also show QA/RQA accuracy using our 1280 annotations in Figure 6, which has the same trend as our metrics.

## 3 Evaluation of QA and RQA

Having designed our tasks (SS2), this section tests llms abilities in QA and RQA. llms struggle in RQA on numerical entities (SS3.1) but surprisingly can often detect their own errors (SS3.2). We study the types of entities that lead to RQA errors (SS3.3) and qualitatively analyze differences between accurate and inaccurate questions from RQA (SS3.4).

### llms Struggle with Numerical Rqa

We first see if RQA (**red**, no stripe) or QA (**blue**, striped) is consistently harder for llms (Figure 2). In numerical domains (Number, Number+Text), llms are much more accurate in QA versus RQA, revealing a clear abduction weakness. Interestingly, in text domains (Easy, Hard), the trend reverses--RQA slightly beats QA 31/32 times. Thus, llms cannot be categorized as always stronger in generation or validation West et al. (2023); Li et al. (2024a): their abilities are domain-specific. If users (e.g. teachers) want to write question-answer pairs with llms, we advise manually writing questions for numerical pairs and answers for text pairs, and using llms to generate the counterparts, given their strengths in numerical QA and textual RQA.

The Numbers domain has the largest QA/RQA accuracy gaps, over 0.8 for LLaMA and Command-R. Some view llms as strong math reasoners, but they excel just in deductive QA tasks, as QA is the main testbed for math abilities Ahn et al. (2024). In contrast, abduction in textual domains appears in instruction-tuning datasets with queries like "Tell me about Germany". Thus, researchers should design more _abductive_ math benchmarks, like RQA, to holistically evaluate llm math capabilities.

### Qa Can Self-Verify Numerical Rqa

We chain RQA and QA for consistency, i.e., see if \(\text{QA}(\text{RQA}(a))\approx a\) (Figure 1). If the check fails, the RQA question \(\hat{q}\) is invalid, the llm fails to answer its own valid \(\hat{q}\), or both failures occur. We discuss how we disentangle these cases below.

llms give: 1) a question \(\hat{q}\) with answer \(a\); and 2) an answer \(\hat{a}\) to their own \(\hat{q}\) without using \(a\). We find three yes/no judgments via our metrics (SS2.3): a) does \(a\) answer \(\hat{q}\) (RQA succeeds); b) does \(\hat{a}\) answer

Figure 2: LLM RQA (**blue**) and QA (**red**) accuracy with 95% CIs for metric error rate. LLMs are much weaker in _abductive_ RQA in numerical settings (Number/Number+Text), but in text settings (Easy/Hard Entity), _deductive_ QA is slightly weaker.

 

\(\hat{q}\) (QA succeeds); and c) are \(a\) and \(\hat{a}\) equivalent? Answers \(\mathcal{A}=(\mathrm{a,b,c})\) to these judgments form a truth table to diagnose llm inconsistencies, which in 91% of cases, fall into the four cases of \(\mathcal{A}\) below:

1. \((\mathrm{y,y,y})\): RQA = QA (consistent).
2. \((\mathrm{n,y,n})\): Just RQA fails.
3. \((\mathrm{y,n,n})\): Just QA fails.
4. \((\mathrm{n,n,n})\): RQA _and_ QA fail.

Other rare cases of \(\mathcal{A}\) are metric prediction errors or errors in \(\hat{q}\) (e.g. ambiguity), which we omit for this analysis. Appendix A.7 shows all cases of \(\mathcal{A}\).

lims are fairly consistent in textual domains, but often fail the check in numerical domains, except GPT-4o (Figure 3, left). Thus, our llMs are logically inconsistent in numerical abduction and deduction. In such cases, QA rarely fails alone: either both RQA and QA fail, where the llm gives an invalid question that it cannot answer, or just RQA fails, where the llm detects its error. The latter is akin to hallucination snowballing Zhang et al. (2024)--inaccurate questions are not just due to knowledge gaps, as llm can answer their invalid question accurately (e.g. 33% of cases for Opus).

For instance, given the answer "127 countries", Opus incorrectly produces the question "How many countries are members of the United Nations that do not have veto power in the UN Security Council?". However, when Opus answers its own question, it knows there are 193 countries in the UN and five of them have veto power,5 returning the correct answer of "188". Thus, self-verification Weng et al. (2023) could be a useful way to verify the correctness of responses in numerical RQA tasks.

Footnote 5: At the time of writing this paper.

### Number+Text RQA Errs on Rare Entities

To find when llms fail in numerical RQA (SS3.1), we test two indicators of RQA error. We first see how often \(a\) appears in the Dolma pretraining corpus Soldaini et al. (2024) via infini-gram Liu et al. (2024), a proxy for the size of all valid questions an llm must abductively reason over in RQA. Next, as SS3.2 hints llms may give overly-hard questions (RQA+QA fail), we use the Prometheus llm Kim et al. (2024) to get a 1-5 difficulty score for \(\hat{q}\). We average metrics pivoted by RQA success/failure on the subset containing human annotations (SS2.3).

Number+Text \(a\) have lower Dolma token counts when RQA fails (Fig 4), so llms struggle to recall long-tail numerical facts Kandpal et al. (2023). In Numbers, RQA \(\hat{q}\) are harder when RQA fails. Thus, calibrating llms with desired difficulty Srivastava and Goodman (2021) could help designers avoid errors from overly-hard questions in RQA on numbers. Also, difficulty and token count are similar in RQA success/failure for Numbers+Text and Numbers, respectively, so RQA errors depend on answer type, like in QA Vakulenko et al. (2020).

### llms Fail to Write Multi-Step Questions

For qualitative insights into question types \(\hat{q}\) from RQA, we analyze 30 \(\hat{q}\) when RQA fails/succeeds in strong llms with low RQA accuracy (SS3.1): L-70B, GPT-4, and Opus. For brevity, we just study the Numbers split, as its similar answers yield \(\hat{q}\) with similar patterns, and group \(\hat{q}\) as: 1) **Single-Step**: has one math operation; 2) **Multi-Step**: has 2+ math operations; 3) **Fact-Based**: tests factual knowledge; and 4) **Metric Error**: metric misclassification. In Appendix A.8, we analyze more questions \(\hat{q}\) in aspects like question novelty, answerability, similarity across models, and memorization.

Figure 4: Answer answer token count in Dolma and question difficulty of when RQA succeeds/fails, averaged over llms.

Figure 3: Logical consistency of RQA and QA. For Number and Number+Text entities, most llms lack consistency (except GPT-4o), with RQA as the main failure point. Otherwise, llms are fairly consistent, with QA as the failure point for Hard Factual Entities. We display the strongest llm from each model family for brevity, with all results shown in Appendix A.7.

 

When RQA fails, \(\hat{q}\) is often multi-step (Fig 5)--combining math and facts (_how many legs are on a human, cat, & spider?_) or adding primes (Fig 1). In contrast, valid \(\hat{q}\) are often single-step (_what is \(19^{2}\)?_) or factual McCarthy (1959) (_how many days is a leap year?_ for _366_). We believe the errors in multi-step RQA are from preference tuning; users favor a complex output even if it is wrong Wen et al. (2025). Thus, curbing complexity bias in alignment, or multi-hop QA decoding methods Zhao et al. (2021), may improve llms in multi-step RQA.

## 4 Related Work

**lllm Reasoning:** Several works have explored lllm reasoning to improve accuracy Qiao et al. (2023) or explainability Si et al. (2024). More recently, works explore if llms can execute diverse reasoning strategies, including inductive Bowen et al. (2024); Yang et al. (2024), deductive Sanyal et al. (2022); Mondorf and Plank (2024), and abductive Zhao et al. (2023); Balepur et al. (2024b) reasoning. However, we are the first to pinpoint abduction abilities via RQA, which differs from traditional question generation setups as we do not have access to an input context Zhang et al. (2021).

**lllm Consistency:**llms must be consistent to reliably help users Visani et al. (2022), but llms are inconsistent under perturbations like prompt format Solar et al. (2024a), entity reversal Berglund et al. (2024), negation Ravichander et al. (2022); Balepur et al. (2024), and ordering Zheng et al. (2024). Recent work finds inconsistencies in lllm generation and verification in math, QA, style transfer, and coding Li et al. (2024a); Gu et al. (2024), which we reproduce via an RQA/QA consistency check. Deb et al. (2023) and Yu et al. (2024) similarly compare llMs in forwards (QA) and backwards (filling question blanks for an answer) reasoning in math. While Deb et al. (2023) claim backwards reasoning is abductive, we argue it is deductive as there is just one answer; we more aptly test abduction/deduction consistency via RQA/QA.

## 5 Conclusion

We test llm RQA and QA abilities. llms have notably low accuracy in numerical RQA which is not just due to knowledge gaps, as models can often answer their own invalid questions correctly. These weaknesses can be excised in future benchmarks to more holistically evaluate lllm numerical abductive reasoning and math capabilities. To reduce inaccuracies in numerical RQA, often from generating overly-complex questions, we suggest calibrating models using difficulty scores, collecting user preferences that control for complexity bias, and adapting prior multi-hop qa methods--key steps for reliable lllm reasoning in downstream tasks.

## 6 Limitations

llms are sensitive to prompt formats Sclar et al. (2024b), so varying prompts could impact lllm accuracy in RQA and QA. To ensure our prompts are reliable, we followed best practices Schulhoff et al. (2024) and kept refining prompts as lllm errors surfaced; the full prompt engineering process is documented in Appendix A.2. Our final prompts will be released and are considered very reasonable implementations of RQA and QA. Further, in Appendix A.6, we test if common prompt engineering strategies (few-shot exemplars, chain-of-thought) can alleviate the low numerical RQA accuracy of GPT-4 but find minimal benefits, suggesting that accuracy gaps between QA and RQA cannot be attributed to prompt formatting alone.

## 7 Ethical Considerations

RQA uses abduction, a core reasoning strategy that aims to arrive at a plausible explanation given a set of facts. However, our current findings suggest that lllm abductive reasoning in numerical settings is highly unreliable. We advise practitioners to take caution when using llms to reason via numerical abduction in downstream tasks, including designing math exam questions, explaining financial forecasts, proposing economic policies, or diagnosing medical patients from numerical data.

## Acknowledgements

We would like to thank the clip lab at University of Maryland and our external collaborators for their feedback, including Neha Srikanth, Haozhe An, Yu Hou, Abhilasah Sanchei, Connor Baumler, Seraphina Goldfarb-Tarrant, Aidan Peppin, Jack

Figure 5: Analysis of Number RQA errors. RQA often fails when the llml tries to give a complex, multi-step question.

 

Wang, and Vishakh Padmakumar. This material is based upon work supported by the National Science Foundation under Grant No. 19-2403436 (BoydGraber), 18-2339746 (Rudinger), 2024-25134012 (Ravichander), and 2024-2236417 (Balepur). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. Cloud computing resources were made possible by a gift from Adobe Research. Access to Command-R was made possible with a Cohere for AI Research Grant.

## References

* Abe (1998) Akinori Abe. 1998. Applications of abduction. In Proc. of ECAI'98 Workshop on Abduction and Induction in AI, pp. 12-19. Citeseer.
* Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 technical report. _arXiv preprint arXiv:2303.08774_.
* Ahn et al. (2024) Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024. Large language models for mathematical reasoning: Progresses and challenges. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, St. Julian's, Malta. Association for Computational Linguistics.
* Anthropic (2023) Anthropic. 2023. Meet claude. [https://www.anthropic.com/product](https://www.anthropic.com/product). Accessed: 2024-09-10.
* Balepur et al. (2024a) Nishant Balepur, Shramay Palta, and Rachel Rudinger. 2024a. It's not easy being wrong: Large language models struggle with process of elimination reasoning. In Findings of the Association for Computational Linguistics ACL 2024, pages 10143-10166, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
* Balepur et al. (2024b) Nishant Balepur, Abhilasha Ravichander, and Rachel Rudinger. 2024b. Artifacts or abduction: How do LLMs answer multiple-choice questions without the question? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10308-10330, Bangkok, Thailand. Association for Computational Linguistics.
* Berglund et al. (2024) Lukas Berglund, Meg Tong, Maximilian Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korrbak, and Owain Evans. 2024. The reversal curse: LLMs trained on "a is b" fail to learn "b is a". In The Twelfth International Conference on Learning Representations, Cited by: SS1, SS2.

 

* Ishihara (2023) Shotaro Ishihara. 2023. Training data extraction from pre-trained language models: A survey. In Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023), Toronto, Canada. Association for Computational Linguistics.
* Jiang et al. (2024) Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Miktal of experts. _arXiv preprint arXiv:2401.04088_.
* Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, T. J. Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zachary Dodds, Nova DasSarma, Eli TranJohnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, John Kermion, Shauna Havcee, Llaine Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom B. Brown, Jack Clark, Nicholas Joseph, Benjamin Mann, Sam McCandlish, Christopher Olah, and Jared Kaplan. 2022. Language models (mostly) know what they know. _ArXiv_, abs/2207.05221.
* Kandpal et al. (2023) Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning, pp. 15696-15707. PMLR.
* Khattab et al. (2024) Omar Khattab, Arnav Singhyi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Srivardhananan A, Saiful Haj, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2024. DSPy: Compiling declarative language model calls into state-of-the-art pipelines. In The Twelfth International Conference on Learning Representations.
* Kim et al. (2023a) Gangwoo Kim, Sungdong Kim, Byeongguk Jeon, Joonsuk Park, and Jaewoo Kang. 2023a. Tree of clarifications: Answering ambiguous questions with retrieval-augmented large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 996-1009, Singapore. Association for Computational Linguistics.
* Kim et al. (2023b) Najoung Kim, Phu Mon Hutt, Samuel R. Bowman, and Jackson Petty. 2023b. (QA)\({}^{2}\): Question answering with questionable assumptions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8466-8487, Toronto, Canada. Association for Computational Linguistics.
* Kim et al. (2024) Seungone Kim, Jamin Shin, Yejin Cho, JoeJ Jang, Shayne Longpre, Huaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2024. Prometheus: Inducing fine-grained evaluation capability in language models. In The Twelfth International Conference on Learning Representations.
* Lia Li et al. (2024a) Xiang Lisa Li, Vaishnavi Shrivastava, Siyan Li, Tatsunori Hashimoto, and Percy Liang. 2024a. Benchmarking and improving generator-validator consistency of language models. In The Twelfth International Conference on Learning Representations.
* Li et al. (2024b) Zongxia Li, Ishani Mondal, Yijun Liang, Huy Nghiem, and Jordan Lee Boyd-Graber. 2024b. Pedants: Cheap but effective and interpretable answer equivalence.
* Liu et al. (2024a) Emmy Liu, Graham Neubig, and Jacob Andreas. 2024a. An incomplete loop: Instruction inference, instruction following, and in-context learning in language models. In First Conference on Language Modeling.
* Liu et al. (2024c) Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. 2024b. Infini-gram: Scaling unbounded n-gram language models to a trillion tokens. In First Conference on Language Modeling.
* Liu et al. (2024c) Michael Xieyang Liu, Frederick Liu, Alexander J Fiannaca, Terry Koo, Lucas Dixon, Michael Terry, and Carrie J Cai. 2024c. "We need structured output": Towards user-centered constraints on large language model output. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pages 1-9.
* McCarthy (1959) John McCarthy. 1959. Programs with common sense.
* Merrill et al. (2024) William Merrill, Noah A. Smith, and Yanai Etzavar. 2024. Evaluating \(n\)-gram novelty of language models using rusty-DAWG. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 14459-14473, Miami, Florida, USA. Association for Computational Linguistics.
* Min et al. (2020) Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigaOA: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5783-5797, Online. Association for Computational Linguistics.
* Mirzadeh et al. (2025) Seyed Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokh, Oncel Tuzel, Samy Bengio, and Mehrdd Farajtabar. 2025. GSM-symbolic: Understanding the limitations of mathematical reasoning in large language models. In The Thirteenth International Conference on Learning Representations.
* Mondorf and Plank (2024) Philipp Mondorf and Barbara Plank. 2024. Comparing inferential strategies of humans and large language models in deductive reasoning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9370-9402, Bangkok, Thailand. Association for Computational Linguistics.
* Pan et al. (2024) Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. 2024. Automatically correcting large language models: Surveying the landscape of diverse automated correction strategies. Transactions of the Association for Computational Linguistics, 12:484-506 

Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen. 2023. Reasoning with language model prompting: A survey. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 5368-5393, Toronto, Canada. Association for Computational Linguistics.
* Ravichander et al. (2022) Abhilasha Ravichander, Matt Gardner, and Ana Marsovic. 2022. CONDAQA: A contrastive reading comprehension dataset for reasoning about negation. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 8729-8755, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Reiter (1989) Raymond Reiter. 1989. Deductive question-answering on relational data bases. In _Readings in Artificial Intelligence and Databases_, pages 431-443. Elsevier.
* Rodriguez et al. (2019) Pedro Rodriguez, Shi Feng, Mohit Iyyer, He He, and Jordan L. Boyd-Graber. 2019. Quizbowl: The case for incremental question answering. _ArXiv_, abs/1904.04792.
* Rogers et al. (2023) Anna Rogers, Matt Gardner, and Isabelle Augenstein. 2023. Qa dataset explosion: A taxonomy of NLP resources for question answering and reading comprehension. _ACM Computing Surveys_, 55(10):1-45.
* Sanyal et al. (2022) Soumya Sanyal, Harman Singh, and Xiang Ren. 2022. FaiRR: Faithful and robust deductive reasoning over natural language. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1075-1093, Dublin, Ireland. Association for Computational Linguistics.
* Schulhoff et al. (2024)Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, Ayush Gupta, HyoJung Han, Sevien Schulhoff, et al. 2024. The prompt report: A systematic survey of prompting techniques. _arXiv preprint arXiv:2406.06608_.
* Selar et al. (2024a) Melanie Selar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024a. Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In _The Twelfth International Conference on Learning Representations_.
* Selar et al. (2024b) Melanie Selar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024b. Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In _The Twelfth International Conference on Learning Representations_.
* except when they are convincingly wrong. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 1459-1474, Mexico City, Mexico. Association for Computational Linguistics.
* Soldaini et al. (2024) Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandau, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Walsh, Luke Zettlemoyer, Noah Smith, Hannanel Hajishirzi, Iz Beltagy, Dirc Groeneveld, Jesse Dodge, and Kyle Lo. 2024. Dolma: an open corpus of three trillion tokens for language model pretraining research. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 15725-15788, Bangkok, Thailand. Association for Computational Linguistics.
* Srivastava and Goodman (2021) Megha Srivastava and Noah Goodman. 2021. Question generation for adaptive education. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_, pages 692-701, Online. Association for Computational Linguistics.
* Vakulenko et al. (2020) Svitlana Vakulenko, Shayne Longpre, Zhucheng Tu, and Raivteja Anantha. 2020. A wrong answer or a wrong question? an intricate relationship between question reformulation and answer selection in conversational question answering. In _Proceedings of the 5th International Workshop on Search-Oriented Conversational AI (SCAI)_, pages 7-16, Online. Association for Computational Linguistics.
* Verga et al. (2024) Pat Verga, Sebastian Hofstadter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. Replacing judges with juries: Evaluating llm generations with a panel of diverse models. _ArXiv_, abs/2404.18796.
* Visani et al. (2022) Giorgio Visani, Enrico Bagli, Federico Chesani, Alessandro Poluzzi, and Davide Capuzzo. 2022. Statistical stability indices for lime: Obtaining reliable explanations for machine learning models. _Journal of the Operational Research Society_, 73(1):91-101.
* Wang et al. (2018) Zichao Wang, Andrew S Lan, Weili Nie, Andrew E Waters, Phillip J Grimaldi, and Richard G Baraniuk. 2018. QG-net: a data-driven question generation model for educational content. In _Proceedings of the fifth annual ACM conference on learning at scale_, pages 1-10.
* Wen et al. (2025) Jiaxin Wen, Ruiqi Zhong, Akbir Khan, Ethan Perez, Jacob Steinhardt, Minlie Huang, Samuel R. Bowman, He He, and Shi Feng. 2025. Language models learn to mislead humans via RLHF. In _The Thirteenth International Conference on Learning Representations_ 

Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. 2023. Large language models are better reasoners with self-verification. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 2550-2575, Singapore. Association for Computational Linguistics.
* West et al. (2023) Peter West, Ximing Lu, Nouha Dziri, Faeze Brahman, Linjie Li, Jena D Hwang, Liwei Jiang, Jillian Fisher, Abhilasa Ravichander, Khyathi Chandu, et al. 2023. The generative AI paradox:"What it can create, it may not understand". In _The Twelfth International Conference on Learning Representations_.
* Yang et al. (2024) Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao, and Furu Wei. 2024. Language models as inductive reasons. In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 209-225, St. Julian's, Malta. Association for Computational Linguistics.
* Young et al. (2024) Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. 2024. Yi: Open foundation models by 01. AI. _arXiv preprint arXiv:2403.04652_.
* Yu et al. (2024) Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2024. Metamath: Bootstrap your own mathematical questions for large language models. In _The Twelfth International Conference on Learning Representations_.
* Yu et al. (2023) Xinyan Yu, Sewon Min, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. CREPE: Open-domain question answering with false presuppositions. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 10457-10480, Toronto, Canada. Association for Computational Linguistics.
* Zhang et al. (2024) Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A. Smith. 2024. How language model hallucinations can snowball. In _Forty-first International Conference on Machine Learning_.
* Zhang et al. (2021) Ruqing Zhang, Jiafeng Guo, Lu Chen, Yixing Fan, and Xueqi Cheng. 2021. A review on question generation from natural language text. _ACM Transactions on Information Systems (TOIS)_, 40(1):1-43.
* Zhao et al. (2021) Chen Zhao, Chenyan Xiong, Jordan Boyd-Graber, and Hal Daume III. 2021. Multi-step reasoning over unstructured text with beam dense retrieval. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4635-4641, Online. Association for Computational Linguistics.
* Zhao et al. (2023) Wenting Zhao, Justin Chiu, Claire Cardie, and Alexander Rush. 2023. Abluctive commonsense reasoning exploiting mutually exclusive explanations. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 14883-14896, Toronto, Canada. Association for Computational Linguistics.
* Zheng et al. (2024) Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024. Large language models are not robust multiple choice selectors. In _The Twelfth International Conference on Learning Representations_ 

Appendix

### Dataset Details

We show details for our dataset in Table 2. Our entities are derived from Quizbowl questions Rodriguez et al. (2019) from the qb Reader api,6 which is free to use and publicly available online. We verify that all questions are answerable by the given answer via Google search. If any question was found to be unanswerable, we manually edited the question such that it was answerable. Thus, all of our collected data is within their license and terms of use, and our use of these questions are within their intended use. Since expert trivia writers curated these questions for academic competitions, we did not need to check that our data has PII. All questions and answers are in English.

Footnote 6: [https://www.qbreader.org/api-docs/](https://www.qbreader.org/api-docs/)

### Prompting Details

Below, we document our prompt engineering process for the QA and RQA prompts shown in Figure 1. To assess each prompt version, we ran inference on a small subset of examples with the Yi and LLaMA llms and manually assessed the quality of questions/answers to identify prevalent issues that could be avoided through prompt engineering. In all adjacent prompt boxes below, blue text corresponds to us adding instructions to the previous version of the prompt, and red-text corresponds to us removing instructions from the previous version.

Our initial RQA prompt is in Prompt A.1. With this prompt, our llms generated verbose answers, so we added the instruction that all questions must be "one-sentence" (Prompt A.2). Next, we observed that it was difficult to reliably parse the question from the model's generated output, so we added formatting constraints (Prompt A.3). At this point, when we looked at the model's generated questions more closely, we saw that models could cheat--adding the answer in the question itself (e.g. giving the question "How many of the 150 people attended the conference" for the answer "150 people"). Thus, we added an instruction to forbid this behavior (Prompt A.4). Finally, as we noticed many of the questions were inaccurate, we wanted to study if abstention could alleviate these issues, so we added an instruction (Prompt A.5) allowing the model to respond with "IDK" SS2.2. We added abstention to test llm calibration Feng et al. (2024), but abstention rates are only 3% in qa and <1% in RQA, so we do not study it in this work. We keep abstention to avoid re-running all llms and omit rare cases of abstention. Our final RQA prompt is in Prompt A.6.

We then designed our QA prompt by mimicking the format of the final RQA prompt, shown in Prompt A.7. We initially wrote the constraint that the answer must be "short" and "just a few words," but we felt these instructions were ambiguous, and the easy and hard entities split of our dataset had answers that were longer than just a few words; as a result, we removed these instructions, and used "the" instead of "a" to make it clear that there is only one valid answer (Prompt A.8). After removing these instructions, we noticed that models would often generate very long explanations before or after answering the question. To avoid this, we added an instruction stating that we were just looking for the answer and no explanation (Prompt A.8). Our final QA prompt is in Prompt A.10.

### Model Details

The llMs used in this work are from the following endpoints:

* LLaMA-8B:Meta-Llama-3-8B-Instruct
* LLaMA-70B: Meta-Llama-3-70B-Instruct
* Mistral-7B: Mistral-7B-Instruct-v0.3
* Mistral-8x7B: Mixtral-8x7B-Instruct-v0.1
* Mistral-8x22B: Mixtral-8x22B-Instruct-v0.1
* Yi-6B:Yi-1.5-6B-Chat
* Yi-9B:Yi-1.5-9B-Chat
* Yi-3dB:Yi-1.5-3dB-Chat
* Command-R:command-r
* Command-R+:command-r-plus
* GPT-3.5:gpt-3.5-turbo-0125
* GPT-4:gpt-4-turbo-2024-04-09
* GPT-4o:gpt-4o-2024-05-13
* Haiku:claude-3-haiku-20240307
* Sonnet:claude-3-sonnet-20240229
* Opus:claude-3-opus-20240229

LLaMA, Mistral, and Yi models are accessed via huggingface, and all other models are accessed through their respective api endpoints. We allocated 8 NVIDIAA:A6000s for Mikxtal-8x22B, 8 NVIDIA:A5000s for Mixtral-8x7B, Yi-3dB, and LLAaM-70B, 2 NVIDIA:A6000s for Vi-9B and LLAaM-8B, and 1 NVIDIA:A6000 for all other 

non-api models (which were run on CPU only). Each model was allocated 24 hours to run both QA and RQA on our dataset.

llms generate with 0 temperature, a minimum token length of 5, and a maximum token length of 5. All other unspecified parameters are set to their respective default values.

### Metric Details

To design a metric for QA accuracy, we consider seven answer equivalence metrics, which check if a candidate answer \(a_{cand}\) is semantically equivalent to a ground-truth answer \(a_{true}\): 1) DSPy-optimized GPT-4o; 2) A rule-based method designed specifically for each dataset; 3) Exact match; 4) Token F1 score; 5) Token Recall Score; 6) Token Precision Score; and 7) PEDANTS Li et al. (2024b), a classifier designed for answer equivalence. The DSPy method in (1) uses a maximum of 10 bootstrapped demos, a maximum of 10 labeled demos, and 20 candidate programs; it uses 64 examples for training (seeding the prompts) and 64 examples for validation. We decide the optimal decision thresholds for (4), (5), and (6) using the 64 validation examples. We present the agreement with human annotations of each metric in Table 3, which is how we picked the metric to use for each dataset split. In all, our QA accuracy metric has 94% raw agreement with humans on 1152 held-out examples.

Since there are no automated metrics to check whether a question \(q\) can correctly be answered by an entity \(a\), we design our own metric for RQA accuracy. Given the strength of the DSPy GPT-4o approach in QA accuracy, we similarly design a DSPy-optimized GPT-4o classifier that determines if \(q\) is correctly answered by \(a\), using the same hyperparameters for QA accuracy. Overall, this RQA accuracy metric has 90% raw agreement with humans on 1152 held-out examples. We also considered Jury approaches Verga et al. (2024), which ensemble multiple llms instead of relying just on a single llm. However, using majority vote with three/five llms boosted our metric's accuracy by less than 2%, which we did not feel justified the much larger computational expenses.

All metrics are reported for a single run, and we provide confidence intervals in Figure 2 corresponding to the error rates in our metrics.

### Abduction/Deduction Human Accuracy

In Figure 6, we show a version of Figure 2 using our human annotations on a subset of data versus the automated metrics on the entire splits. Our trend holds on the human-annotated subset; llms are still much weaker in numerical RQA versus QA, but their QA capabilities slightly beat RQA in the text-based settings.

### RQA with Prompting Engineering

To explore if RQA weaknesses can be alleviated with prompt engineering efforts Schulhoff et al. (2024), we test three prompting strategies: 1) Zero-Shot Chain-of-Thought Prompting (asking the llm to "Think step by step" before answering); 2) Self-Verification (asking the llm to "Check if the question is accurate after generating a question"); and 3) Five-Shot Prompting (including five exemplars showing the model how to generate a question for an answer). To write exemplars for (3), we pick question/answer pairs when RQA succeeds in the zero-shot setting to make the priors in the exemplars most similar to the model's original generations. The prompts for (1), (2), and (3) are in Prompts A.11, A.12, and A.13, respectively.

We experiment with GPT-4 on Numbers and Numbers+Text, as the model showed a surprising RQA weakness in these settings. GPT-4 is also considered to respond well to prompt engineering efforts, making it a suitable candidate for our prompting strategies. Overall, none of these prompting strategies can close the accuracy gap between RQA and QA (Figure 7). Chain-of-thought prompting increases GPT-4's RQA accuracy by \(\sim 0.15\), but it is still significantly lower than QA, which does not use chain-of-thought. This shows that the accuracy gap between QA and RQA may be an inherent reasoning flaw of current llms that cannot be fully mitigated via prompt engineering.

### Full Consistency Analysis

In this section, we describe the consistency analysis for all values of our truth table \(\mathcal{A}\), introduced in SS3.2. Apart from the four categories described before, the truth table outcome can also be "Ambiguous Question" if \(\mathcal{A}=(\mathtt{y},\mathtt{y},\mathtt{n})\), as both steps succeeded but converged to different answers (meaning the question had more than one possible correct answer). Another option is for the mistakes to cancel out, which is a rare scenario \(\mathcal{A}=(\mathtt{n},\mathtt{n},\mathtt{y})\) where the model generated an inaccurate question and answered its own question incorrectly, but managed to arrive at the original entity \(a\). The final category is a Metric Prediction Error, a scenario that only occurs if either just QA or RQA was 

predicted to fail, but \(a\) and \(\hat{a}\) were predicted to be matching (\(\mathcal{A}=(n,y,y)\) or \(\mathcal{A}=(y,n,y)\)). These scenarios are summarized in Table 4.

Figure 8 reports the full consistency analysis for all 16 of our llms and all truth table scenarios. The four categories reported in Figure 3 encompass most of the truth table. Further, even for smaller lms, our claims hold; llms can often detect their own question inaccuracies from RQA through QA.

### Further Analysis of RQA Questions

Due to page limit constraints of a short paper, we were unable to show the entire qualitative analysis we conducted on questions generated in RQA. Below, we give more qualitative results on the answerability of questions from RQA (Appendix A.8.1), a cross-model comparison of question duplicates in RQA (Appendix A.8.2), the ability of llms to match the ground-truth question during RQA (Appendix A.8.3), and a brief investigation into memorization in the RQA task (Appendix A.8.4).

#### a.8.1 Are RQA questions unanswerable?

We now seek to understand the types of RQA questions generated in the Number+Text setting, complementing our analysis in SS3.4. The Number+Text questions have higher variance and cannot be as neatly categorized as in SS3.4 (e.g. single-step computation). So instead, we study the _answerability_ of 30 generated questions from each llm, i.e., if the question is clear but leads to an incorrect answer, or if the question has an issue that makes it difficult to answer. We adopt five categories of unanswerable questions from Rogers et al. (2023):

1) **Invalid Premise:** the question contains a false assumption, so it is impossible to answer. For example, Opus generates the question _How old was the world's oldest tortoise, Jonathan, when he passed away in 2022?_, but this Tortoise is still alive.

2) **No Consensus on the Answer:** the question does not have a single, agreed-upon answer . For example, LLaMA generates the question _What is the unique property of the Lie algebra E8 that makes it particularly interesting in theoretical physics?_, but Lie algebra has many distinct, interesting properties that would answer the question.

3) **Information not yet Discovered:** the answer to this question is not yet known. For example, GPT-4 generates the question _How long, in terms of word count, is the sentence that holds the record for being the longest in the English language without using any punctuation?_, but it is not yet known what could theoretically be the longest sentence.

4) **Missing Information:** the question does not have enough information, or it is too vague. For example, GPT-4 generates the question _How many individuals attended the annual community festival last year according to the final headcount?_, which cannot be answered without knowing more details.

5) **Answerable:** The question has one right answer.

As expected, when RQA succeeds, questions are mostly answerable (Figure 9). However, a non-trivial proportion of generated questions when RQA fails are unanswerable, reaching nearly 60% for GPT-4. The most common types of unanswerable questions are those that are missing information, meaning that they are too vague or ambiguous, or those that have false premises or assumptions. While several works explore methods to _answer_ ambiguous questions (Min et al., 2020; Kim et al., 2023a) or questions with false presuppositions (Yu et al., 2023; Kim et al., 2023b), our analysis reveals a need to _avoid generating_ ambiguous or faulty-presupposition questions in RQA.

In Tables 5 and 6, we provide examples of question/error types in our qualitative analysis on the Number and Number+Text split, respectively.

#### a.8.2 Do llms give the same RQA questions?

While most of our analysis treated llms independently, we now study whether llms generate the same exact questions (i.e. duplicates) in RQA. Figure 11 shows that llms more frequently generate duplicated questions across entities versus matching questions from other models. For example, LLaMA-3 70B generates 379 duplicate questions in the Numbers setting, even when the input answer is altered. This aligns with very recent work suggesting that llms may often conduct pattern-matching rather than engaging in true, generalizable reasoning (Mirzadeh et al., 2025).

Interestingly, models in the same family are more likely to generate duplicated questions. For example, GPT-3.5, GPT-4, and GPT-4o generate the same questions in RQA more often than when compared to other llm families. Thus, we speculate that these model families likely share similar pre-training and alignment data, which is optimized on through different training recipes.

#### a.8.3 Does RQA match the gold question?

We now explore whether the questions generated for an answer in RQA match the gold question we collected for that answer. When determining if 

the two questions are semantically equivalent, we follow the protocol of Balepur et al. (2024) and analyze whether the two questions test the exact same knowledge. Figure 10 shows that the llms can often match the true question when RQA succeeds in Number+Text settings, reaching as high as 40% of cases for GPT-4; the questions never matched for Number. One explanation for the high match rate is dataset contamination Ishihara (2023), but it is also possible that the most likely question the llm inductively reason towards is the ground-truth question. For example, for the answer "120 counties," the only salient fact linked to the entity is that Kentucky has 120 counties McCarthy (1959); this led GPT-4's question and the ground-truth question to both ask about Kentucky.

#### a.8.4 Are any RPA questions memorized?

Since the duplicates in Appendix A.8.3 suggest that llms may just be retrieving similar questions from pretraining rather than reasoning towards new questions in RQA, we now investigate the _novelty_ of the RQA questions Merrill et al. (2024), i.e., whether they are exactly copied from pretraining. We do not know which corpora all of our llms are trained on, so we use the Dolma Soldaini et al. (2024) corpus as a proxy for pretraining data. For each generated RQA question \(\hat{q}\), we compute how frequently the exact question \(\hat{q}\) appears in Dolma via infini-gram Liu et al. (2024).

Table 7 reveals in total, \(2.87\)% of RQA questions are exactly found in Dolma. For comparison, \(1.25\)% of our ground-truth questions exist in Dolma. While we did not explicitly prompt the model to give a new question that it has not seen in pretraining, practitioners may need to design specialized techniques if they desire novel RQA questions.

When comparing exact question match frequency by model, weaker/smaller llms tend to copy more from pretraining data, suggesting that smaller llms are more prone to RQA memorization. Further, the Hard Fact setting is much less prone to question copying in RQA, likely because the RQA input answers have very low pretraining token count (SS3.3), which further supports that llms may struggle to retrieve exact pretraining knowledge for long-tail facts Kandpal et al. (2023).

We present examples of RQA questions that appear the most in Dolma in Table 8. The tendency to generate inaccurate or ambiguous questions may be influenced by pretraining, as many of these questions appear directly in Dolma 

* [10] A. K. S. G. Wilson, and M. J. W. York, "The quantum theory of the classical field theory: The role of a new quantum theory in physics," _Phys. Rev. D_**58**(6):2379-2402, 1998.
* [11] R. C. Myers and P. H. Davies, "Quantum mechanics for quantum fields with applications to quantum gravity," _Journal of Physics B_**40**(1):103-118, 1984.
* [12] L. F. Abbott, "A general introduction to quantum field theories," _Class. Quant. Grav._**2**(1):1231-1246, 1999.
* [13] E. Witten, "String Theory and Quantum Gravity," _Nucl. Phys. Proc. Suppl._**B108**(1):1-16, 1984.
* [14] T. Banks, "On the string world interpretation of black hole entropy," _Phys. Lett. B_**108**(1):1-16, 1984.
* [15] N. Arkani-Hamed, S. Dimopoulos, and I. Antoniadis, "Non-perturbative corrections to the string world description," _Phys. Lett. B_**108**(1):1-16, 1984.
* [16] N. Arkani-Hamed, S. Dimopoulos, and I. Antoniadis, "Anomalous string dual to the string world description," _Phys. Lett. B_**108**(1):1-16, 1984.
* [17] N. Arkani-Hamed, S. Dimopoulos, and I. Antoniadis, "Dynamics of the superstring sector," _Phys. Lett. B_**108**(1):1-16, 1984.
* [18] N. Arkani-Hamed, S. Dimopoulos, and I. Antoniadis, "Supergravity duality from string theory," _Phys. Lett. B_**108**(1):1-16, 1984.
* [19] N. Arkani-Hamed, S. Dimopoulos, and I. Antoniadis, "Supersymmetric strings at large distances," _Phys. Lett. B_**108**(1):1-16, 1984.

[MISSING_PAGE_POST]

 



[MISSING_PAGE_POST]

 



[MISSING_PAGE_POST]

 



\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & **Number** & **Number+Text** & **Easy Entity** & **Hard Entity** \\ \hline Count & 900 & 743 & 900 & 900 \\ Average Answer Length (Tokens) & 1.00 & 2.49 & 2.77 & 5.18 \\ Average Question Length (Tokens) & 8.75 & 21.9 & 18.9 & 22.9 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Dataset details of each split (Number, Number+Text, Easy Entity, Hard Entity), including the number of data instances, average length of answers (in tokens), and average length of questions (in tokens). Tokens are computed using tiktoken.

Figure 6: llm deduction (**blue**) and abduction (**red**) accuracy based on human annotations on a subset of data (20 labels per model/dataset). The plot shows a similar trend as the automated metrics (llms are weaker in abduction in numerical settings, but stronger in abduction in non-numerical settings), confirming the validity of our metrics.

Figure 7: llm deduction (**blue**) and abduction (**red**) accuracy with GPT-4 on numerical entities. For QA, we present the zero-shot prompt used in §3.1. For RQA, we test adding chain-of-thought instructions (GPT-4 + CoT), asking the llm to verify its question post-generation (GPT-4 + Self-Verification), and including five exemplars (GPT-4 + 5-Shot). None of these strategies allow the model to fully match the QA accuracy.





\begin{table}
\begin{tabular}{c c c c} Is \(a_{true}\) the answer to \(q_{bowd}?\) & Is \(a_{bwd}\) the answer to \(q_{bwd}?\) & Is \(a_{true}\) equal to \(a_{bwd}?\) & **Outcome** \\ \hline Yes & Yes & Yes & RQA = QA \\ Yes & Yes & No & Ambiguous Question \\ Yes & No & Yes & QA Fails \\ Yes & No & No & Metric Error (Impossible) \\ No & Yes & Yes & RQA Fails \\ No & Yes & No & Metric Error (impossible) \\ No & No & Yes & RQA + QA Fail \\ No & No & No & Mistakes Cancel (lucky!) \\ \end{tabular}
\end{table}
Table 4: All truth table outcomes for the consistency analysis in §3.2.

Figure 8: QA and RQA logical consistency across all models. The consistency trends are also prevalent for smaller/less capable llms; RQA and QA consistency is higher for easy/hard entities, but lms can often detect their own RQA inaccuracies in numerical settings.

Figure 9: Error analysis of questions from RQA on Number+Text. When RPA fails, questions are often unanswerable (30-60%), and frequently include false premises or omit key information that is needed to answer the question.




\begin{table}
\begin{tabular}{l l c l} \hline \hline
**Question** & **Answer** & **Model** & **Error Type** \\ \hline How many British soldiers were killed or wounded during the Battle of Thermopylae in 480 BCE? & 266 men & L-70B & Invalid Premise \\ What is the numerical designation..., if we humorously assume there were 111 before it? & 112 Afk & GPT-4 & Invalid Premise \\ According to a 2011 census, how many officially recognized ethnic groups are there in India? & 53 d distinct peoples & Opus & Invalid Premise \\ In what year did the Vietnamese seeing Le Ideon defeat the Song Dynasty army at the Battle of Bach Dang? & 988 AD & L-70B & No Consensus \\ How long did the construction of the Great Wall of China continue...? & 264 years & GPT-4 & No Consensus \\ What is the wavelength of yellow light in the visible spectrum? & 587 nanometers & L-70B & Missing Info \\ How many individuals attended the annual community festival last year according to the final headcount? & 178 people & GP7+4 & Missing Info \\ How old was the world’s oldest tortoise, Jonathan, when be passed away in 2022? & 179 years of age & Opus & Missing Info \\ \hline \hline \end{tabular}
\end{table}
Table 6: Examples of RQA question types and errors on the Number+Text split.

Figure 10: Proportion of RQA questions on Numbers+Text that semantically match the ground-truth question when RQA succeeds. LLaMA-3 70B, GPT-4, and Opus can all match the ground-truth question over 25% of the time.

 



\begin{table}
\begin{tabular}{l c c c c|c} \hline \hline Model & Easy Fact & Hard Fact & Number & Number+Text & **Model Sum** \\ \hline Mis-7b & 25 & 6 & 6 & 7 & 44 \\ Mix-7B & 7 & 0 & 10 & 1 & 18 \\ Mix-22B & 41 & 3 & 17 & 12 & 73 \\ Yi-6B & 18 & 0 & 98 & 40 & 156 \\ Yi-9B & 17 & 2 & 8 & 1 & 28 \\ Yi-34B & 7 & 0 & 18 & 0 & 25 \\ L-8B & 12 & 2 & 21 & 8 & 43 \\ L-70B & 4 & 1 & 19 & 4 & 28 \\ Command-R & 48 & 3 & 61 & 47 & 159 \\ Command-R+ & 28 & 3 & 17 & 20 & 68 \\ GPT-3.5 & 111 & 19 & 16 & 105 & 251 \\ GPT-4 & 29 & 0 & 32 & 3 & 64 \\ GPT-4o & 96 & 10 & 27 & 39 & 172 \\ Haiku & 88 & 12 & 67 & 95 & 262 \\ Sonnet & 51 & 5 & 18 & 17 & 91 \\ \hline Opus & 41 & 0 & 48 & 12 & 101 \\
**Dataset Sum** & 623 & 66 & 483 & 411 & **1583** \\ \hline \hline \end{tabular}
\end{table}
Table 7: Number of generated RQA questions that are exact matches to a question in the Dolma pretraining corpus. On average, models are prone to copying questions from pretraining \(\sim 3\)% of the time. Smaller/weaker llms are more susceptible to copying questions from pretraining in RQA. Further, easy facts and numerical answers are more likely to lead to copied questions in RQA versus our hard facts.

Figure 11: Cross-model frequency of questions from RQA that are exact duplicates. llms often generate the same question in RQA even though the input answer changes, reaching as high as 379 for LaMA-3 70B on Numbers.





\begin{table}
\begin{tabular}{l l l l l l}
**Question** & **Answer** & **Model(s)** & **Split** & **Valid** & **Count** \\ What is the answer to this question? & Lucy poems & Haiku & Hard Fact & No & 21313 \\ Who lives in a pineapple under the sea? & Spongebob Squarepants & GPT-3.5, GPT-4o & Easy Fact & Yes & 1452 \\ Where does the story take place? & In the Penal Colony & GPT-3.5 & Hard Fact & No & 1395 \\ How many countries are there in the world? & 195 nations & GPT-3.5 & Num+Text & Yes & 380 \\ What is the capital of France? & Paris, France & Command-R+ & Easy Fact & Yes & 338 \\ Who was the first president of the United States? & George Washington & Haiku, Sonnet & Easy Fact & Yes & 281 \\ How many days are there in a week? & 357 & Yi-6B & Number & No & 194 \\ What is the capital of the United States? & Washington, D.C. & Command-R, GPT-3.5 & Easy Fact & Yes & 192 \\ How many days are there in a year? & 365 & Command-R & Easy Fact & Yes & 166 \\ How many days are there in a year? & 800 & Haiku & Number & No & 166 \\  & & & & \\ \end{tabular}
\end{table}
Table 8: Questions generated from RQA that are most frequently found in the Dolma corpus. The llm’s tendency to generate inaccurate questions (e.g. _How many days are there in a year?_ for _800_) or ambiguous questions (_What is the answer to this question?_) could be influenced by how often these questions appear in pretraining.