PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in
  Action
Authors: name: Yijia Shao, name: Tianshi Li, name: Weiyan Shi, name: Yanchen Liu, name: Diyi Yang

As language models (LMs) are widely utilized in personalized communication
scenarios (e.g., sending emails, writing social media posts) and endowed with a
certain level of agency, ensuring they act in accordance with the contextual
privacy norms becomes increasingly critical. However, quantifying the privacy
norm awareness of LMs and the emerging privacy risk in LM-mediated
communication is challenging due to (1) the contextual and long-tailed nature
of privacy-sensitive cases, and (2) the lack of evaluation approaches that
capture realistic application scenarios. To address these challenges, we
propose PrivacyLens, a novel framework designed to extend privacy-sensitive
seeds into expressive vignettes and further into agent trajectories, enabling
multi-level evaluation of privacy leakage in LM agents' actions. We instantiate
PrivacyLens with a collection of privacy norms grounded in privacy literature
and crowdsourced seeds. Using this dataset, we reveal a discrepancy between LM
performance in answering probing questions and their actual behavior when
executing user instructions in an agent setup. State-of-the-art LMs, like GPT-4
and Llama-3-70B, leak sensitive information in 25.68% and 38.69% of cases, even
when prompted with privacy-enhancing instructions. We also demonstrate the
dynamic nature of PrivacyLens by extending each seed into multiple trajectories
to red-team LM privacy leakage risk. Dataset and code are available at
https://github.com/SALT-NLP/PrivacyLens.