Title: X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Benchmark: X-FACTR
Summary: Language models have effectively captured factual knowledge through cloze-style fill-in-the-blank questions, but evaluations have mostly focused on English. To assess factual knowledge retrieval across different languages, a multilingual benchmark for cloze-style probes covering 23 diverse languages is created, along with expanded methods and decoding algorithms for multi-word entities. The study also introduces a code-switching method to enhance multilingual models' knowledge access, demonstrating its effectiveness across several languages.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Factual knowledge retrieval
Phenomenon defined by authors: Yes
Phenomenon definition: factual knowledge retrieval in LMs in different languages than English
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: The cloze-style prompts used therein are manually created and consist of a sequence of
tokens, where [X] and [Y] are placeholders for subjects and objects (e.g. “[X] is a [Y] by profession.”). To assess the existence of a certain fact, [X] is replaced with the actual subject and the model predicts the object in the blank
Task item definition: subject, object/fact, answer, scores
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: >500,000 facts
Dataset metadata available: Yes
Dataset metadata detail: language, percentage in dataset
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum
Metric subscores: Yes
Metric subscores detail: language, independence, order, confidence
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Knowledge / General
Alternate taxonomy: Multilinguality
New taxonomy: General Application / General Knowledge