Title: Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection
Summary: The paper tests the robustness of instruction following LMMs to prompt injection when the context maybe injected by adversaries, i.e. LLM driven web search.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: robustness, instruction following
Phenomenon defined by authors: No
Phenomenon definition: robustness of LLMs in following instructions against prompt injection attacks
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: The task is question answering when part of the context provided to the model is injected with deviating instructions.
Task item definition: Question + Context (inkl injected deviating instruction) + Answer
Task source: Modified from another benchmark (e.g. translation into another language)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Unknown
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean across samples, then the relative decline in accuracy due to injection is computed.
Metric subscores: Yes
Metric subscores detail: Metrics are provided for each adapted dataset.
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Industry
Benchmark availability: Yes
Task face validity: yes, but only a small subset
Metric face validity: Yes
Results interpreted by authors: No
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The authors acknowldege the problems of underlying datasetsand the very limited scope of the "task and dataset" used in this work.
Taxonomy: Instruction Following / 
New taxonomy: General Capability / Instruction Following