Title: What Are the Odds? Language Models Are Capable of Probabilistic Reasoning
Summary: Attempt to evaluate probablistic reasoning capabilities of LLMs. Do so by asking LLMs perform basic probability questions on common probability distributions a handful of real world distributions.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Probablistic reasoning 
Phenomenon defined by authors: No
Phenomenon definition: They losely define probablistic reasoning as "A form of numerical reasoning that is important for interpreting many different forms of data is contextualizing an individual measurement or measurements (a sample or samples) within a population (a distribution)." 

Also: "Thinking probabilistically is efficient as one does not have to represent every detail of every sample that one observes, and instead can have the data summarized with a small number of parameters that describe the distribution (Lindskog et al., 2021)."
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Purpose notes: Sub-elements are "Estimating percentiles", "Drawing samples" and "Calculating probabilities"
Task definition: Define 3 sub-tasks: 1) "Estimating probabilities: Given a distribution, estimate the percentile a ssample is in" 2) Drawing samples: Given a distribution model is asked to draw samples from it 3) Cacluating probabilities: Given a distribution estimate the probability a sample will fall between two given values.
Task item definition: Consider the following parameters that describe a normal distribution:
Mean: 43.20
Standard Deviation: 30.50

What is the percentile of the value 35 within the provided distribution?
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 3 tasks for each of 5 common distributions, 3 real world ones. 
Dataset size detail: Test set is extremely small
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Split by real world vs toy distributions. Broken down by area. 
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Industry
Benchmark availability: No, no link is provided
Task face validity: Probablistic reasoning is a wide ranging and difficult to estimate phenomenon, and whilst these tasks do measure a subset of this phenomenon they don't come close to measuring everything.
Metric face validity: Yes
Results interpreted by authors: No
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Reasoning / Mathematical
New taxonomy: General Capability / Mathematical Reasoning