Omkar2118/datasets huggingface dataset:
Inferential-Strategies

A collection of large languge model responses to tasks of propositional logic. The responses are annotated according to the following criteria:

- Inferential strategy employed by the model. Strategies considered are: *supposition following*, *chain construction*, *compound strategy*, *concatenation strategy*, and the *symbolic strategy*. Binary labels are assigned to each strategy, indicating whether the strategy is present in the model's response.
- Assessment of the validity of the model's final conclusion. Binary labels are assigned to each response, indicating whether the model's conclusion is accurate ("valid_conclusion").
- Evaluation of the soundness of the model's rationale. Binary labels are assigned to each response, indicating whether the rationale provided by the model is sound ("sound_reasoning").
- A description of the model's reasoning error. This information is provided in the form of a string ("reasoning_errors").

Introducing paper: Comparing Inferential Strategies of Humans and Large Language Models in Deductive Reasoning
TruthQuest

A benchmark for suppositional reasoning based on the principles of knights and knaves puzzles. Knights and knaves problems represent a classic genre of logical puzzles where characters either tell the truth or lie. The objective is to logically deduce each character's identity based on their statements. The challenge arises from the truth-telling or lying behavior, which influences the logical implications of each statement.

*TruthQuest* encompasses problems of varying complexity, considering both the number of characters and the types of logical statements involved. The dataset comprises 2,400 instances. Each instance has a unique solution.

Introducing paper: Liar, Liar, Logical Mire: A Benchmark for Suppositional Reasoning in Large Language Models