Title: MULTITRUST: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Models
Benchmark: MULTITRUST
Summary: The paper introduces a benchmark on the trustworthiness of MLLMs across five primarcy aspects: truthfulness, safety, robustness, fairness, and privacy. It benchmarks 20+ MLMMs and highlights the complexities introduced by multi-modality. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: trustworthiness
Phenomenon defined by authors: Yes
Phenomenon definition: "Drawing on extensive studies in trustworthy LLMs and distilling from relevant literature of MLLMs, we pinpoint 5 primary aspects of trustworthiness for evaluating MLLMs, including truthfulness, safety, robustness, fairness, and privacy. In particular, truthfulness, safety, and robustness guarantee the models’ reliability and stability in preventing undesirable outcomes, i.e., errors, harms, and variations under different conditions" (page 3)
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: There are 32 tasks that are generative and/or discriminative, multimodal or test-only. Wide range from NSFW image description to PII leakage in conversations They utilize off the shelf datasets, augment existing ones, and create their own. 
Task item definition: Varies greatly from task to task. Some include images and a prompt, others just a prompt. 
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions)
Dataset size: "more than 15k" (page 103)
Dataset metadata available: Yes
Dataset metadata detail: image information, types of queries
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Unknown
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics), Distribution (perplexity, calibration, correlation), Correlation (Matthew's correlation, Pearson's r)
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum, correlation between overall rankings and general capabilities based on MMBench
Metric subscores: Yes
Metric subscores detail: In the main body, it's sub-aspect (e.g., truthfulness, safety, robustness). In the appendix, there are subsets of many of the tasks/existing benchmarks they run.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Alignment / Safety
New taxonomy: General Capability / Safety