RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Reversing the Question Answering Task
Question answering ( QA ) is a long-standing task in NLP Green Jr et al. ( 1961 ) .
Given an input question q , QA aims to deduce the correct answer a Reiter ( 1989 ) .
More recently, large language models (LLMs) have been used for the inverse task—given an answer a , generate a valid question q to which a is the answer—which we call reverse question answering ( RQA ).We distinguish from question generation Zhang et al. ( 2021 ) , which uses an input context to ground the answer. RQA is used in downstream tasks like exam generation Biancini et al. ( 2024 ) and brainstorming Xu et al. ( 2024 ) .

QA and RQA are often tested separately, but we test them jointly, offering two key benefits. First , it gives insights into open questions on LLM abilities, as some show LLMs excel in generation over comprehension (West et al., 2023 , RQA ) , while others claim verification is easier (Kadavath et al., 2022 , QA ) .
Uncovering which task is harder can guide benchmark design Chen et al. ( 2024 ) and inform data collection practices in writing question-answer pairs ( section 3.1 ; e.g., if RQA is easy, get answers manually and then generate synthetic questions).

Second , chaining RQA and QA forms a consistency check for LLM reasoning Liu et al. ( 2024a ) . RQA —inferring just one of many valid questions—is abductive Abe ( 1998 ) , while QA —inferring an answer from question premises—is deductive Reiter ( 1989 ) .
Thus, by seeing if \textsc{QA}(\textsc{RQA}(a))\approx a , i.e., checking if an LLM can answer its own question from RQA (Fig 1 ), we can assess LLMs’ logical robustness in abduction and deduction ( section 3.2 ).
This analysis can also help determine if LLMs can reliably self-verify Pan et al. ( 2024 ) in downstream RQA tasks like writing exams Wang et al. ( 2018 ) or proposing research questions Schurz ( 2008 ) .

To reap these benefits, we test if 16 LLMs can produce 1) questions correctly answered by input entities ( RQA ); and 2) accurate answers for input questions ( QA ).
We collect 3443 trivia question/answer pairs Rodriguez et al. ( 2019 ) , grouped by answer as either numerical or textual entities, forming inputs to evaluate RQA and QA in varied domains.

In numerical domains, LLMs are much less accurate in RQA than QA , especially integers (Fig 1 ); the accuracy difference when LLMs do these tasks exceed 0.80 for Command-R and LLaMA-3 ( section 3.1 ).
Interestingly, in textual domains, the trend reverses, so LLMs are not consistently better generators or validators Li et al. ( 2024a ) .
We then design a consistency check ( section 3.2 ) to see if LLMs can answer their own RQA questions; numerical RQA inaccuracies are not solely due to knowledge gaps, as LLMs often answer their own invalid questions correctly in QA (33% of cases for Claude-Opus).
We then analyze questions from RQA ( section 3.3 , section 3.4 ) and find RQA errors occur when LLMs give overly-complex, multi-step questions, giving insights into strategies—like complexity bias mitigation in preference data and calibrating models using difficulty scores—to improve LLM reliability for RQA .
Experimental Setup
We evaluate LLM abilities in question answering ( QA ) and reverse question answering ( RQA ): 1) \textsc{QA}(q)\rightarrow\hat{a} : Given a question q with a single answer a , the LLM gives an answer \hat{a} for q . QA succeeds if a matches \hat{a} semantically. This typical QA setup tests deduction , as the model must reason to the correct answer a based on premises in q . 2) \textsc{RQA}(a)\rightarrow\hat{q} : Given answer a , the LLM must give a question \hat{q} . RQA succeeds if the correct answer to \hat{q} is a (verified via oracle, section 2.3 ). RQA tests abduction , as the model must reason toward one of many valid questions with the answer of a .

Below, we describe the datasets ( section 2.1 ), models ( section 2.2 ), and metrics ( section 2.3 ) used for RQA and QA .
Dataset Collection
We use 4 domains of question/answer pairs (q,a) for QA and RQA inputs, based on answer type of a (Table 1 ).
We group them as numerical (Number, Number+Text) or textual (Easy Fact, Hard Fact), providing varied domains to test QA and RQA .

For a of type (1), q is a random, one-step math operation (what is 118+211?).
Other types are from QANTA Rodriguez et al. ( 2019 ) , an expert-curated dataset of multi-sentence trivia QA pairs.
For (3)–(4), a is the answer to sampled QANTA questions, with the last sentence as q .
We use middle school questions for (3) and college questions for (4) to discern between easy/hard facts.
We find type (2) a in QANTA via regex and q from the sentence a appears in.
We verify all QA pairs (Appendix A.1 ).
Models
We test 16 LLMs: GPT (Achiam et al., 2023 , 3.5, 4, 4o) , Command R (Cohere, 2023 , Command-R, Command-R+) , Claude (Anthropic, 2023 , Sonnet, Haiku, Opus) , LLaMA-3 Instruct (Dubey et al., 2024 , 8B, 70B) , Yi-1.5 Chat (Young et al., 2024 , 6B, 9B, 34B) , and Mistral Instruct (Jiang et al., 2024 , 7B, 8x7B, 8x22B) . All use 0 temperature.

The QA and RQA prompts are 0-shot, as few-shot exemplars test inductive reasoning, not deduction/abduction Liu et al. ( 2024a ) in QA / RQA .
Exemplars also do not improve LLM RQA accuracy (Appendix A.6 ).
Prompts follow the same template as Figure 1 with format rules to parse outputs Liu et al. ( 2024c ) .
Two NLP graduate students write the prompts, with all design steps in Appendix A.2 .
Evaluation Metrics
To compute QA accuracy, two graduate students researching QA annotate if 1280 LLM QA answers \hat{a} for a question q match its true answer a (20 per answer type/model).
We test seven metrics Li et al. ( 2024b ) that check if \hat{a} and a are equivalent.
We select DSPy-optimized Khattab et al. ( 2024 ) GPT-4o for easy/hard entities and a rule-based method for numerical entities, since these methods had the highest agreement with humans (94% on average).

For RQA accuracy, students annotate if the answer to 1280 questions \hat{q} from RQA is a (20 per answer type/model), following rules from Li et al. ( 2024b ) .
We use DSPy-optimized GPT-4o as an oracle ( \textsc{Verify}^{*}(\hat{q},a) ) to verify if a is the answer to \hat{q} , which has high ( 90\% ) human agreement.
Metric agreement is high but imperfect, so we also show QA / RQA accuracy via our 1280 annotations (Figure 6 ), which has the same trend as our metrics.
LLMs Struggle with Numerical RQA
We first see if RQA ( red , no stripe) or QA ( blue , striped) is consistently harder for LLMs (Figure 2 ).
In numerical domains (Number, Number+Text), LLMs are much more accurate in QA versus RQA , revealing a clear abduction weakness.
Interestingly, in text domains (Easy, Hard), the trend reverses— RQA slightly beats QA 31/32 times.
Thus, LLMs cannot be categorized as always stronger in generation or validation West et al. ( 2023 ); Li et al. ( 2024a ) : their abilities are domain-specific.
If users (e.g. teachers) want to write question-answer pairs with LLMs, we advise manually writing questions for numerical pairs and answers for text pairs, and using LLMs to generate the counterparts, given their strengths in numerical QA and text RQA .

The Numbers domain has the largest QA / RQA accuracy gaps, over 0.8 for LLaMA and Command-R.
Some view LLMs as strong math reasoners, but they excel just in deductive QA tasks, as QA is the main testbed for math abilities Ahn et al. ( 2024 ) .
In contrast, abduction in textual domains appears in instruction-tuning datasets with queries like “Tell me about Germany.”
Thus, researchers should design more abductive math benchmarks, like RQA , to holistically evaluate LLM math capabilities.
QA Can Self-Verify Numerical RQA
We chain RQA and QA for consistency, i.e., see if \textsc{QA}(\textsc{AQ}(a))\approx a (Fig 1 ). If the check fails, the RQA question \hat{q} is invalid, the LLM fails to answer its own valid \hat{q} , or both fail, disentangled below.

LLMs produce: 1) a question \hat{q} with the correct answer of a ; and 2) an answer \hat{a} to their own \hat{q} without seeing a .
We study three yes/no queries via our metrics ( section 2.3 ): a) is a the answer to \hat{q} ( RQA succeeds); b) is \hat{a} the answer to \hat{q} ( QA succeeds); and c) are a and \hat{a} equivalent?
Answers \mathcal{A}=(\text{a},\text{b},\text{c}) to these queries form a truth table to diagnose inconsistencies, which in 91% of cases, fall into four types of \mathcal{A} :
1) (\texttt{y},\texttt{y},\texttt{y}) : RQA = QA (consistent);
2) (\texttt{n},\texttt{y},\texttt{n}) : just RQA fails;
3) (\texttt{y},\texttt{n},\texttt{n}) : just QA fails;
4) (\texttt{n},\texttt{n},\texttt{n}) : RQA and QA fail.
Other rare cases of \mathcal{A} are metric prediction errors or special properties of \hat{q} (e.g. ambiguity), which we omit for this analysis.
Appendix A.7 shows all cases of \mathcal{A} .

LLMs are fairly consistent in textual domains, but often fail the check in numerical domains, except GPT-4o (Figure 3 , left).
Thus, our LLMs are logically inconsistent in numerical abduction and deduction.
In such cases, QA rarely fails alone: either both RQA and QA fail, where the LLM gives an invalid question that it cannot answer, or just RQA fails, where the LLM detects its error.
The latter is akin to hallucination snowballing Zhang et al. ( 2023 ) —inaccurate questions are not just due to knowledge gaps, as the model can still answer its invalid question accurately (e.g. 33% of cases for Opus).
Thus, self-verification Weng et al. ( 2023 ) could be a useful strategy to verify and improve the correctness of outputs in numerical RQA tasks.
Number+Text RQA Errs on Rare Entities
To see when LLMs fail in numerical RQA ( section 3.1 ), we test two indicators of RQA error.
We first see how often a appears in the Dolma pretraining corpus Soldaini et al. ( 2024 ) via infini-gram Liu et al. ( 2024b ) , a proxy for the size of all valid questions an LLM must abductively reason over in RQA .
Next, section 3.2 hints LLMs may give overly-hard questions ( QA + AQ fail), so we use a Prometheus LLM Kim et al. ( 2024 ) to get a 1-5 difficulty score for \hat{q} .
We average metrics pivoted by RQA success/failure.

Number+Text a have lower Dolma token counts when RQA fails (Fig 4 ), so LLMs struggle to recall long-tail numerical facts Kandpal et al. ( 2023 ) .
In Numbers, RQA \hat{q} are harder when RQA fails.
Thus, calibrating LLMs with desired difficulty Srivastava and Goodman ( 2021 ) could help designers avoid errors from overly-hard questions in RQA on numbers.
Also, difficulty and token count are similar in RQA success/failure for Numbers+Text and Numbers, respectively, so RQA errors depend on answer type, like in QA Vakulenko et al. ( 2020 ) .
LLMs Fail to Write Multi-Step Questions
For qualitative insights into question types \hat{q} from RQA , we analyze 30 \hat{q} when RQA fails/succeeds in strong LLMs with low RQA accuracy ( section 3.1 ): L-70B, GPT-4, and Opus.
Due to page limits, we use the Numbers split, as its similar answers yield \hat{q} with similar patterns, and group \hat{q} as: 1) Single-Step : has one math operation; 2) Multi-Step : has 2+ math operations; 3) Fact-Based : tests factual knowledge; and 4) Metric Error : metric misclassification.
In Appendix A.8 , we analyze more questions \hat{q} in aspects like question novelty, answerability, similarity across models, and memorization.

When RQA fails, \hat{q} is often multi-step (Fig 5 )—combining math and facts ( how many legs are on a human, cat, & spider? ) or adding primes (Fig 1 ).
In contrast, valid \hat{q} are often single-step ( what is 19^{2} ? ) or factual McCarthy ( 1959 ) ( how many days is a leap year? for 366 ).
We believe the habit of multi-step RQA is from preference tuning; users favor a complex output even if it is wrong Wen et al. ( 2024 ) .
Thus, curbing complexity bias in alignment, or multi-hop QA decoding methods Zhao et al. ( 2021 ) , may improve LLMs in multi-step RQA .
Related Work
LLM Reasoning: Prior works study LLM reasoning to improve accuracy Qiao et al. ( 2023 ) or explainability Si et al. ( 2024 ) .
More recently, works see if LLMs can execute diverse reasoning types, such as inductive Bowen et al. ( 2024 ); Yang et al. ( 2024 ) , deductive Sanyal et al. ( 2022 ); Mondorf and Plank ( 2024 ) , and abductive Zhao et al. ( 2023 ); Balepur et al. ( 2024b ) reasoning.
However, we are the first to target abduction via RQA , which differs from typical question generation setups as we have no access to an input context Zhang et al. ( 2021 ) .

LLM Consistency: LLMs must be consistent to reliably help users Visani et al. ( 2022 ) , but LLMs are inconsistent under perturbations like prompt format Sclar et al. ( 2024 ) , entity reversal Berglund et al. ( 2024 ) , negation Ravichander et al. ( 2022 ); Balepur et al. ( 2024a ) , and option order Zheng et al. ( 2024 ) .
Recent work finds inconsistencies in LLM generation and verification Li et al. ( 2024a ) , which we reproduce in our QA / RQA consistency check.
Similarly, Deb et al. ( 2023 ) and Yu et al. ( 2024 ) compare LLMs in forwards (QA) and backwards (filling question blanks given an answer) reasoning in math.
While Deb et al. ( 2023 ) claim that backwards reasoning is abductive, we argue it is deductive as there is just one answer; we more aptly test abduction/deduction consistency via RQA / QA .
Conclusion
We test LLM RQA and QA abilities.
LLMs have notably low accuracy in numerical RQA which is not just due to knowledge gaps, as models can often answer their own invalid questions correctly.
These weaknesses can be excised in future benchmarks to more holistically evaluate LLM numerical abductive reasoning and math capabilities.
To reduce inaccuracies in numerical RQA , often from generating overly-complex questions, we suggest calibrating models using difficulty scores, collecting user preferences that control for complexity bias, and adapting prior multi-hop QA methods—key steps for reliable LLM reasoning in downstream tasks.
Limitations
LLMs are sensitive to prompt formats Sclar et al. ( 2023 ) , so varying prompts could impact LLM accuracy in RQA and QA .
To ensure our prompts are reliable, we followed best practices Schulhoff et al. ( 2024 ) and kept refining prompts as LLM errors surfaced; the full prompt engineering process is documented in Appendix A.2 .
Our final prompts will be released and are considered very reasonable implementations of RQA and QA .
Further, in Appendix A.6 , we test if common prompt engineering strategies (few-shot exemplars, chain-of-thought) can alleviate the low numerical RQA accuracy of GPT-4 but find minimal benefits, suggesting that accuracy gaps between QA and RQA cannot be attributed to prompt formatting alone.
Ethical Considerations
RQA uses abduction, a core reasoning strategy that arrives at a plausible explanation given a set of facts.
However, our current findings suggest that LLM abductive reasoning in numerical settings is highly unreliable.
We advise practitioners to take caution when using LLMs to perform numerical abductive reasoning in downstream tasks, including designing math exam questions, explaining financial forecasts, proposing economic policies, or diagnosing medical patients from numerical data.
Acknowledgements
We would like to thank the CLIP lab at University of Maryland and our external collaborators for their feedback, including Neha Srikanth, Jack Wang, Vishakh Padmakumar, Haozhe An, Yu (Hope) Hou, Abhilasha Sancheti, Seraphina Goldfarb-Tarrant, Aidan Peppin, and Connor Baumler.
This material is based upon work supported by the National Science Foundation under Grant No. iis -2403436 (Boyd-Graber), iis -2339746 (Rudinger), and dge -2236417 (Balepur).
Any opinions, findings, and conclusions or recommendations expressed
in this material are those of the author(s) and do not necessarily
reflect the views of the National Science Foundation.
Cloud computing resources were made possible by a gift from Adobe Research.
Access to Command-R was made possible through the Cohere for AI Research Grant Program.
A.1 Dataset Details
We show details for our dataset in Table 2 .
Our entities are derived from Quizbowl questions Rodriguez et al. ( 2019 ) from the QB Reader API 2 2 2 https://www.qbreader.org/api-docs/ , which is free to use and publicly available online.
We verify that all questions are answerable by the given answer via Google search.
If any question was found to be unanswerable, we manually edited the question such that it was answerable.
Thus, all of our collected data is within their license and terms of use, and our use of these questions are within their intended use.
Since expert trivia writers curated these questions for academic competitions, we did not need to check that our data has PII.
All questions and answers are in English.
A.2 Prompting Details
Below, we document our prompt engineering process for the QA and RQA prompts shown in Figure 1 .
To assess each prompt version, we ran inference on a small subset of examples with the Yi and LLaMA LLMs and manually assessed the quality of questions/answers to identify prevalent issues that could be avoided through prompt engineering.
In all adjacent prompt boxes below, blue text corresponds to us adding instructions to the previous version of the prompt, and red text corresponds to us removing instructions from the previous version.

Our initial RQA prompt is in Prompt A.8.4 .
With this prompt, our LLMs generated verbose answers, so we added the instruction that all questions must be “one-sentence” (Prompt A.8.4 ).
Next, we found that it was difficult to reliably parse the question from the model’s generated output, so we added formatting constraints (Prompt A.8.4 ).
At this point, when we looked at the model’s generated questions more closely, we found that models could cheat—adding the answer in the question itself (e.g. giving the question “How many of the 150 people attended the conference” for the answer “150 people”).
Thus, we added an instruction to forbid this behavior (Prompt A.8.4 ).
Finally, as we noticed many of the questions were inaccurate, we wanted to study if abstention could alleviate these issues, so we added an instruction (Prompt A.8.4 ) allowing the model to respond with “IDK” section 2.2 .
We added abstention to test LLM calibration Feng et al. ( 2024 ) , but abstention rates are only 3% in QA and <1% in QG, so we do not study it in this work.
We keep abstention to avoid re-running all LLMs and omit rare cases of abstention.
Our final RQA prompt is shown in Prompt A.8.4 .

We then designed our QA prompt by mimicking the format of the final RQA prompt, shown in Prompt A.8.4 .
We initially wrote the constraint that the answer must be “short” and “just a few words,” but we found these instructions to be ambiguous, and the easy and hard entities split of our dataset had answers that were longer than just a few words; as a result, we removed these instructions, and used “the” instead of “a” to make it clear that there is only one valid answer (Prompt A.8.4 ).
After removing these instructions, we found that models would often generate very long explanations before or after answering the question.
To avoid this, we added an instruction stating that we were just looking for the answer and no explanation (Prompt A.8.4 ).
Our final QA prompt is shown in Prompt A.8.4 .
A.3 Model Details
The LLMs used in this work are from the following endpoints: {itemize*}

LLaMA-8B: Meta-Llama-3-8B-Instruct

LLaMA-70B: Meta-Llama-3-70B-Instruct

Mistral-7B: Mistral-7B-Instruct-v0.3

Mixtral-8x7B: Mixtral-8x7B-Instruct-v0.1

Mixtral-8x22B: Mixtral-8x22B-Instruct-v0.1

Yi-6B: Yi-1.5-6B-Chat

Yi-9B: Yi-1.5-9B-Chat

Yi-34B: Yi-1.5-34B-Chat

Command-R: command-r

Command-R+: command-r-plus

GPT-3.5: gpt-3.5-turbo-0125

GPT-4: gpt-4-turbo-2024-04-09

GPT-4o: gpt-4o-2024-05-13

Haiku: claude-3-haiku-20240307

Sonnet: claude-3-sonnet-20240229

Opus: claude-3-opus-20240229

LLaMA, Mistral, and Yi models are accessed via huggingface, and all other models are accessed through their respective API endpoints.
We allocated 8 NVIDIA:A6000s for Mixtral-8x22B, 8 NVIDIA:A5000s for Mixtral-8x7B, Yi-34B, and LLaMA-70B, 2 NVIDIA:A6000s for Yi-9B and LLaMA-8B, and 1 NVIDIA:A6000 for all other non-API models (which were run on CPU only).
Each model was allocated 24 hours to perform both QA and RQA on our dataset.

LLMs generate with 0 temperature, a minimum token length of 5, and a maximum token length of 5. All other unspecified parameters are set to their respective default values.
A.4 Metric Details
To design a metric for QA accuracy, we consider seven answer equivalence metrics, which check if a candidate answer a_{cand} is semantically equivalent to a ground-truth answer a_{true} : 1) DSPy-optimized GPT-4o; 2) A rule-based method designed specifically for each dataset; 3) Exact match; 4) Token F1 score; 5) Token Recall Score; 6) Token Precision Score; and 7) PEDANTS Li et al. ( 2024b ) , a classifier designed for answer equivalence.
The DSPy method in (1) uses a maximum of 10 bootstrapped demos, a maximum of 10 labeled demos, and 20 candidate programs; it uses 64 examples for training (seeding the prompts) and 64 examples for validation.
We find optimal decision thresholds for (4), (5), and (6) using the 64 validation examples.
We present the agreement with human annotations of each metric in Table 3 , which is how we decided which metric to use for each dataset split.
Overall, our QA accuracy metric has 94% raw agreement with humans on 1152 held-out examples.

Since there are no automated metrics to check whether a question q can correctly be answered by an entity a , we design our own metric for RQA accuracy.
Given the strength of the DSPy GPT-4o approach in QA accuracy, we similarly design a DSPy-optimized GPT-4o classifier that determines if q is correctly answered by a , using the same hyperparameters for QA accuracy.
Overall, this RQA accuracy metric has 90% raw agreement with humans on 1152 held-out examples.
We also considered Jury approaches Verga et al. ( 2024 ) , which ensemble multiple LLMs instead of relying just on a single LLM.
However, we found that using majority vote with three/five LLMs boosted our metric’s accuracy by less than 2%, which we did not feel justified the much larger computational expenses.

All metrics are reported for a single run, and we provide confidence intervals in Figure 2 corresponding to the error rates in our metrics.
A.5 Abduction/Deduction Human Accuracy
In Figure 6 , we show a version of Figure 2 using our human annotations on a subset of data versus the automated metrics on the entire splits.
Our trend holds on the human-annotated subset; LLMs are still much weaker in numerical RQA versus QA , but their QA capabilities slightly beat RQA in the text-based settings.
A.6 RQA with Prompting Engineering
To explore if RQA weaknesses can simply be alleviated with prompt engineering efforts Schulhoff et al. ( 2024 ) , we test three prompting strategies: 1) Zero-Shot Chain-of-Thought Prompting (asking the LLM to “Think step by step” before answering); 2) Self-Verification (asking the LLM to “Check if the question is accurate after generating a question”); and 3) Five-Shot Prompting (including five exemplars showing the model how to generate a question for an answer).
To write exemplars for (3), we pick question/answer pairs when RQA succeeds in the zero-shot setting to try and make the priors in the exemplars most similar to the model’s original generations.
The prompts for (1), (2), and (3) are in Prompts A.8.4 , A.8.4 , and A.8.4 , respectively.

We experiment with GPT-4 on Numbers and Numbers+Text, as the model showed a surprising RQA weakness in these settings. GPT-4 is also considered to respond well to prompt engineering efforts, making it a suitable candidate for our prompting strategies.
Overall, none of these prompting strategies can close the accuracy gap between RQA and QA (Figure 7 ).
Chain-of-thought prompting increases GPT-4’s RQA accuracy by \sim 0.15 , but it is still significantly lower than QA , which does not use chain-of-thought.
This shows that the accuracy gap between QA and RQA may be a fundamental reasoning flaw of current LLMs that cannot be fully mitigated through prompt engineering.