Title: DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models
Benchmark: DecodingTrust
Summary: The paper introduces a comprehensive trustworthiness evaluation for large language
models with a focus on GPT-4 and GPT-3.5. The benchmark introduced considers diverse perspectives including toxicity, stereotype bias, adversarial robustness, out-of-distribution robustness, robustness on adversarial demonstrations, privacy, machine ethics, and fairness. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Trustworthiness
Phenomenon defined by authors: No
Phenomenon definition: "toxicity, stereotype bias, adversarial robustness, out-of-distribution robustness, robustness on adversarial demonstrations, privacy, machine ethics, and fairness." (page 2)
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Presenting LLMs with different scenarios that invoke one of the subsets of trustworthiness (e.g., stereotypical input) and assessing output.
Task item definition: Varies by task (the benchmark includes many tasks and datasets) but seems to include user prompt, system prompt, choices (when prompts are MCQ)
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions)
Dataset size detail: They provide the total # of prompts used for each trustworthiness subset (in the context of calculating computational costs) but since they run a lot of variations (e.g., different system prompts), it's unclear what the size of the bare benchmark is. This is in Appendix K.
Dataset metadata available: Yes
Dataset metadata detail: system prompt, prompt template
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), Distribution (perplexity, calibration, correlation)
Metric aggregation: Simple Mean, Weighted Mean
Metric statistics: simple mean, weighted mean
Metric subscores: Yes
Metric subscores detail: different system prompts, normal vs challenging prompts, different adversarial demonstrations, different adversarial text generation strategies, different demographic groups and stereotype topics, different types of PII, different sensitive attributes
Metric metascoring: pass@k (any correct answer in k trials)
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: indirectly address it
Author validity detail: In the limitations section: "Subjectivity. Trustworthiness perspectives such as toxicity, stereotype bias, machine ethics, and fairness involve subjectivity and should be human-centric in their definitions and evaluations. Here we aim to provide our objective observations, and leave the analysis of how these metrics are aligned with human as an important future work to explore model behaviors based on human understanding"
Taxonomy: Alignment / Alignment
New taxonomy: General Capability / Alignment