Title: JASMINE: Arabic GPT Models for Few-Shot Learning
Summary: The paper introduces a suite of Arabic autoregressive Transformer language models ranging in size and pre-trained on a large and diverse dataset. It also introduces a benchmark for automated and human evaluation of Arabic autoregressive models, with coverage of social biases, harms, and toxicity.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: (broad) Arabic language capabilities 
Phenomenon defined by authors: No
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Not defined
Task definition: There are multiple tasks: autocompletion, commonsense inference, word manipulation, news story generation, poetry generation, dialectal generation
Task item definition: Varies, but mostly prompt + context if needed
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language)
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: For one of the datasets, they say it's a 8:2 train/test split, with test = 1,675
Dataset size notes: For one of the datasets, they say it's a 8:2 train/test split, with train = 14,288 and test = 1,675
Dataset metadata available: Yes
Dataset metadata detail: Different for different datasets; poetry dataset: topics, speech transcription dataset: country/dialect.
Sampling method: Random sample (creators defined a task space and sampled from it), Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Response format: Multiple choice, Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), Human ratings (text quality, preference, NOT manual scoring of other metrics), Distribution (perplexity, calibration, correlation)
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Unclear
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Author validity detail: They discuss in the limitations section the fact that they could not cover some important Arabic dialects. While they don't explicitly discuss this in the context of construct validity, I think it's an important construct validity question.
Taxonomy: NLP / 
Alternate taxonomy: Multilinguality
New taxonomy: General Capability / Bias