Title: The Tail Tells All: Estimating Model-Level Membership Inference Vulnerability Without Reference Models
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: membership inference attacks, privacy, deep learning, memorization

Abstract: Membership inference attacks (MIAs) have emerged as the standard tool for evaluating the privacy risks of AI models. However, state-of-the-art attacks require training numerous, often computationally expensive, reference models, limiting their practicality. We present a novel approach for estimating model-level vulnerability, TPR at low FPR, to membership inference attacks without requiring reference models. Empirical analysis shows loss distributions to asymmetric and heavy-tailed and suggest that most points at risk from MIAs to have moved from the tail (high-loss region) to the head (low-loss region) of the distribution. We leverage this insight to propose a method to estimate model-level vulnerability from the training and testing distribution alone: using the absence of outliers from the high-loss region as a predictor of the risk. We evaluate our method, the TNR of a simple loss attack, across a wide range of architectures and datasets and show it to accurately estimate model-level vulnerability to the SOTA MIA attack (LiRA). We also show our method to outperform both low-cost (few reference models) attacks such as RMIA and other measures of distribution difference. We finally evaluate the use of non-linear function to evaluate risk and show the approach to be promising to evaluate the risk in large-language models.

TL;DR: We introduce a novel method to predict a model's vulnerability to SOTA membership inference attacks without the need for expensive reference model-based approaches while maintaining accurate risk assessment across various architectures and datasets.