Title: Premise Order Matters in Reasoning with Large Language Models
Summary: Benchmark that shows a failure mode of LLM reasoning - if the order of sentences in the question are reversed / permuted, then LLMs suddenly fail to answer questions they could previously answer. They present results for logic and maths.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Reasoning
Phenomenon defined by authors: No
Phenomenon definition: In this work, we investigate the effect that premise order has on LLM reasoning.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Standard reasoning question answer format. Two datasets are used:

1) Logical reasoning, given sets of facts that hold, sets of rules (if A then B) and a conclusion (Cis True). Have to determine whether the conclusion is correct.

2) Maths: GSM8K maths question dataset, but with sentence order changed.
Task item definition: Triplet of (question, permuted order question, answer)

ie: Question: Thomas withdraws $1000 in 20 dollar bills from the bank account. He loses 10 bills while getting home. After that, he uses half of the remaining bills to pay for a bill. Thomas then triples his money. He then converts all his bills to 5 dollar bills. How many 5 dollar bills does he have?

Permuted question: Thomas withdraws $1000 in 20 dollar bills from the bank account. After getting home, he uses half of the remaining bills to pay for a bill.
Thomas then triples his money. He then converts all his bills to 5 dollar bills. He lost 10 bills while getting home. How many 5 dollar bills does he have?

Answer: Thomas has 240 five-dollar bills.
Task source: Modified from another benchmark (e.g. translation into another language)
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 220
Dataset metadata available: Yes
Dataset metadata detail: Kendall tau distance between question and permuted question. Flag indicating whether distractors were added to question. 
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Dataset source (Logic or Maths), Kendal tau distance, Whether distractors used.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Industry
Benchmark availability: Unclear
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Reasoning / Logical
New taxonomy: General Capability / Logical Reasoning
Procedural notes: From Deepmind: They say they release the benchmark, but no link provided and cannot find online.