Title: GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Models
Benchmark: GEOMLAMA
Summary: In this paper, a benchmark dataset is introduced, Geo-diverse Commonsense Multilingual Language Models Analysis (GEOMLAMA), for probing the diversity of the relational knowledge in multilingual PLMs. 
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: geo-diverse commonsense probing
Phenomenon defined by authors: Yes
Phenomenon definition: do PLMs store geo-diverse commonsense knowledge?
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: investigating whether PLMs are capable of predicting correct answers among all the possibilities of different countries.
Task item definition: A prompt and  a gold answer. For each concept,  there are multiple masked multilingual prompts with specified country information [X] querying geo-diverse knowledge about the concept.
Task source: Expert-crafted task examples (e.g. hand-written examples)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 3125
Dataset metadata available: Yes
Dataset metadata detail: Language of the prompt. 
Sampling method: Random sample (creators defined a task space and sampled from it)
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall), Distribution (perplexity, calibration, correlation)
Metric subscores: Yes
Metric subscores detail: By language. 
Metric access: Model access required (e.g. logits)
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Knowledge / Cultural
New taxonomy: General Application / Cultural Knowledge