Keep Security! Benchmarking Security Policy Preservation in Large
  Language Model Contexts Against Indirect Attacks in Question Answering
Authors: name: Hwan Chang; name: Yumin Kim; name: Yonghyun Jun; name: Hwanhee Lee
Published: 2025-05-21 17:58:11

Abstract: As Large Language Models (LLMs) are increasingly deployed in sensitive
domains such as enterprise and government, ensuring that they adhere to
user-defined security policies within context is critical-especially with
respect to information non-disclosure. While prior LLM studies have focused on
general safety and socially sensitive data, large-scale benchmarks for
contextual security preservation against attacks remain lacking. To address
this, we introduce a novel large-scale benchmark dataset, CoPriva, evaluating
LLM adherence to contextual non-disclosure policies in question answering.
Derived from realistic contexts, our dataset includes explicit policies and
queries designed as direct and challenging indirect attacks seeking prohibited
information. We evaluate 10 LLMs on our benchmark and reveal a significant
vulnerability: many models violate user-defined policies and leak sensitive
information. This failure is particularly severe against indirect attacks,
highlighting a critical gap in current LLM safety alignment for sensitive
applications. Our analysis reveals that while models can often identify the
correct answer to a query, they struggle to incorporate policy constraints
during generation. In contrast, they exhibit a partial ability to revise
outputs when explicitly prompted. Our findings underscore the urgent need for
more robust methods to guarantee contextual security.