Title: Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory
Benchmark: CONFAIDE
Summary: The paper introduces a benchmark grounded in the theory of contextual integrity and designed to identify critical weaknesses in the privacy reasoning capabilities of instruction-tuned LLMs. It consists of four tiers, gradually increasing in complexity, with the final tier evaluating contextual privacy reasoning and theory of mind capabilities.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: contextual privacy reasoning
Phenomenon definition: "the appropriate flow of information within specific social contexts. A privacy breach happens when the information flows against the contextual norm." (page 1)
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: There are four tiers and four tasks: 
1. rate how sensitive people would consider certain information to be
2.  judge whether a given information flow scenario aligns with people's privacy expectations
3. generate contextually appropriate responses and understand information accessibility when a secret is shared between two parties, and one of them interacts with a third party
4. create list of action items and meeting summary while excluding sensitive info and containing vital public info
Task item definition: combinations of the following depending on the task: information types, vignettes, relationship between parties, meeting notes
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size detail: Total size of the dataset is not provided but some information is provided in the tasks. For example, task from tier 2 has 98 vignettes, task from tier 3 has 270 scenarios, and task from tier 4 has 20 transcripts.
Dataset metadata available: Yes
Dataset metadata detail: difficulty level (tiers), information type, relationship pairs, incentive, actor, use, secret, public information
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM post-processing (extracting answers, reformatting for automated scoring), Correlation (Matthew's correlation, Pearson's r)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: By tiers (difficulty levels)
Metric metascoring: pass@k (any correct answer in k trials)
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: Indirectly address it
Author validity detail: They link the design of the benchmark directly to an established theory of contextual integrity by drawing on prior empirical research that operationalized this theory. For example, the prompts used in tiers 1 and 2 are "taken directly from Martin & Nissenbaum (2016)" (page 4).
Taxonomy: Alignment / Alignment
New taxonomy: General Capability / Alignment