Title: Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Bias
Benchmark: Cross-Care
Summary: This paper examines how LMs associate disease prevalence with different demographic groups. The authors introduce Cross-Care, a benchmark probing this association across 89 diseases and nine demographic groups. Applying Cross-Care to a series of LMs, the authors find substantial misalignment between LM representation of disease prevalence and real disease prevalence rates across demographic groups.
Phenomenon type: Specific form of bias
Target phenomenon: They want to measure representational biases in LMs, focusing on medical information.
Phenomenon defined by authors: Yes
Phenomenon definition: "the representation of disease prevalence across diverse demographic groups" (p. 1)
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: The task consists of measuring the probability assigned by LMs to sentences associating demographic groups with diseases (e.g., "[DEMOGRAPHIC] patients usually have [DISEASE]").
Task item definition: Each item is a sentence associating a demographic group with a disease.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task source detail: The basis for the benchmark are two dictionaries: a dictionary of demographic terms and a dictionary of diseases. Both are taken from prior resources and, in the latter case, expanded by the authors. The authors then use ten templates that are filled with a demographic term and a disease to yield one item of the benchmark.
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 8,010 for each of four considered languages
Dataset size detail: The benchmark is not released as such; the authors solely release the templates and the dictionaries. The size of the benchmark is computed based on the size of these three components; it is not explicitly mentioned in the paper.
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: The task is not based on responses; it relies solely on the probability assigned to the tokens in the sentence.
Metric definition: Mean of the output logits
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: four languages, different demographic groups
Metric access: Model access required (e.g. logits)
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Computing the mean of the logits does not seem mathematically sound, but the general approach of examining the output probabilities is valid.
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Alignment / Bias
New taxonomy: General Capability / Bias