Title: Do Vision-Language Models Respect Contextual Integrity in Location Disclosure?
Primary area: datasets and benchmarks
Keywords: Benchmarking, NLP datasets, Evaluation Methodologies, Privacy, Geolocation, VLM, Contextual Integrity

Abstract: Vision-language models (VLMs) have recently demonstrated strong performance in image geolocation, identifying images' location to a precision that now surpasses specialized systems. This capability poses a serious privacy threat, as these widely accessible models may enable malicious actors to infer sensitive locations from photos casually shared online. While prior work has proposed applying a blanket restriction on geolocation disclosure to combat this risk, these measures fail to distinguish valid geolocation uses from malicious behavior. Instead, VLMs should maintain contextual integrity by reasoning about elements within an image to determine the appropriate extent of location information to disclose. To evaluate this ability, we introduce a benchmark challenging VLMs to interpret latent social cues in real-world images and determine the appropriate level of location disclosure. Our evaluation of 14 leading VLMs shows that, despite their abilities to precisely geolocate images, models are poorly aligned with human privacy expectations. They often over-disclose in sensitive contexts and are vulnerable to prompt-based attacks. Our results call for new design principles in multimodal systems to incorporate context-conditioned privacy reasoning.

TL;DR: A benchmark for evaluating VLMs' contextual privacy judgment in geolocation tasks.