Title: Can We Infer Confidential Properties of Training Data from LLMs?
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

Data Confidentiality

Property Inference Attack

Privacy
Abstract: Large language models (LLMs) are increasingly fine-tuned on domain-specific datasets to support applications in fields such as healthcare, finance, and law. These fine-tuning datasets often have sensitive and confidential dataset-level properties — such as patient demographics or disease prevalence—that are not intended to be revealed. While prior work has studied property inference attacks on discriminative models (e.g., image classification models) and generative models (e.g., GANs for image data), it remains unclear if such attacks transfer to LLMs. In this work, we introduce PropInfer, a benchmark task for evaluating property inference in LLMs under two fine-tuning paradigms: question-answering and chat-completion. Built on the ChatDoctor dataset, our benchmark includes a range of property types and task configurations. We further propose two tailored attacks: a prompt-based generation attack and a shadow-model attack leveraging word frequency signals. Empirical evaluations across multiple pretrained LLMs show the success of our attacks, revealing a previously unrecognized vulnerability in LLMs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the task of dataset property inference for LLMs. This is specifically targeted at identifying dataset-level confidential information like customer demographics or patient statistics, when LLMs are for instance fine-tuned on domain-specific datasets (healthcare, finance).\n\nAuthors identify that prior work has studied a similar question for other data modalities and models, including tabular data and GANs. They argue that property inference for LLMs introduces two new challenges: \n- Sensitive properties might be more complex\n- LLMs do no fit cleanly into discriminative or generative, make it harder to see what kind of attacks work best\n\nThe paper introduces PropInfer, a new benchmark task for property inference in LLMs. \nThey propose two attacks:\n- Prompting the model to generate synthetic samples from the target LLM, to then aggregate them and use the aggregation as an estimated property ratio. \n- Shadow-model based attack based on word frequency, training a meta-attack model to predict property ratio based on word frequencies. \n\nThey evaluate these attacks in different settings, considering 1 dataset, different properties, multiple pretrained models and two different finetuning methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The paper introduces a novel vulnerability and attack in the context of LLMs, the relevance of which is clearly articulated throughout the work. \n- The dataset considered in the paper very nicely fits the purpose of the attack. \n- Considering both SFT and CLM is interesting to study from a memorization perspective, also in the context of gender (more in the question) and diagnosis (more in the answer). \n- The two attacks proposed seem to work quite well, beating well thought-through baselines. \n- Overall, I really like the problem the paper studies and believe the experiments are adequate evidence for the claims made. \n\n**Weaknesses:** \n- I think the paper could benefit from a bit more related work on other (privacy) attacks leveraging similar techniques. For instance, [1,2] use word frequencies or an n-gram model to run black-box access MIAs, with [2] also doing shadow modeling. While it's quite different, [3] might be useful to. \n- Only one dataset is used to evaluate the methods, but I would not call this a major concern. I would just find it interesting to see if authors find other use-cases where this is problem statement is equally relevant and the attacks are equally successful. \n\t\n[1] Kaneko, M., Ma, Y., Wata, Y., & Okazaki, N. (2024). Sampling-based pseudo-likelihood for membership inference attacks. arXiv preprint arXiv:2404.11262.\n\n[2] Meeus, M., Wutschitz, L., Zanella-Béguelin, S., Tople, S., & Shokri, R. (2025). The Canary's Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text. arXiv preprint arXiv:2502.14921.\n\n[3] Hayase, J., Liu, A., Choi, Y., Oh, S., & Smith, N. A. (2024). Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?. arXiv preprint arXiv:2407.16607.\n\nQUESTIONS:\n- In the perplexity attack baseline, how do you compute the perplexity? \n- I find the analsysis of which keywords are most predictive quite interesting (Appendix A.3). Could you also add the words that are most useful to make the actual prediction in the best performing meta-attack model? It would be nice to confirm that the model learns that when a word like 'female' is more common, the meta attack model would be more likely to predict a higher female patient ratio. \n- For the word frequency attack, do you really need 100k samples? What would happen if you gave the same query budget to the BB attack? \n- Maybe a discussion paragraph would be helpful, touching on what other properties would be useful, potentially beyond privacy too. I'm thinking about language distribution, percent of statements biased towards a certain political topic etc.\n\nLIMITATIONS:\nOnly one dataset, but this is not a major concern.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the dataset property inference task in LLMs. Similar to membership inference, evaluating dataset property inference helps to protect confidential attributes of datasets from leaking. Given lack of property evaluation setup in LLMs, this paper introduces PropInfer which leverages the ChatDoctor dataset to construct two setups matching two LLM fine-tuning paradigms. In addition, the paper illustrates the effectiveness of various attacking methods including a novel one tailored for LLM. The results suggest that property inference is a critical issue in LLMs that requires attention to better defense.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n1. The paper is well motivated. The presentation is clear and effective.\n2. The benchmark setup is convincing, comprehensive and robust.\n3. The result and analysis is reasonable and demonstrates effectiveness.\n\nWeakness (detailed in Questions section below):\n1. As a benchmark, the result section is somewhat limited in terms of variabilities in model architecture, size, etc. (See Q2 below).\n2. While most of the claims and motivations are well-supported. Some important claims can be further clarified (See Q3&4 below).\n\nQUESTIONS:\n1. While I appreciate the Generation w/o FT baseline, it is still unclear what effects pretraining has on the property inference. Particularly for larger models, existing biases (e.g., gender bias in doctor-patient setting) from large-scale pretraining may interfere with property inference evaluation. I would love to see this being discussed if not experimented.\n2. Continuing from (1), the set of models evaluated is limited given the author claims PropInfer as a benchmark. It would be really assertive and make the paper stronger if more models (particular larger ones provided your resource permits) are included in the experiments. It can help clarify influence from large-scale pretraining and showcase generalization of the method to most LLMs.\n3. The author mentioned an intuition on L230-231. However, this hypothesis is not well-tested. Later on L338-339 and in Appx A.3, it also seems that the performance is sensitive to the choice of $d$. This rationale deserves a deeper dive to both understand the paper better and make the method more convincing.\n4. What is the source for the claim on L93-95? Even if the claim is true, it is still worth to add as a baseline and show that it is less effective than proposed attacks.\n5. Appx A.1 mentioned a human study. Could you provide the survey details? E.g., how many participants and are they medical experts or general audience?\n\nI would raise my score if the experiment results can be verified more generally and/or some of the concerns/hypothesis/claims above can be clarified.\n\nLIMITATIONS:\nThere is one limitation which is tied to Question 1 above. Given the social biases in pretrained large-scale models, it may influence the property inference results and mistakenly inherit the bias as dataset property.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies property inference attacks on large language models (LLMs). Specifically, it asks whether adversaries can recover dataset-level attributes rom an LLM fine-tuned on sensitive data. The authors introduce a benchmark task PropInfer, based on the Chat-Doctor medical Q&A dataset. They consider two fine-tuning modes (Q&A and chat completion) and select a range of sensitive properties (e.g. disease rates, gender balance). They propose two attack strategies: (1) a black-box generation-based attack that crafts prompts and measures the frequency of outputs containing the target property; and (2) a shadow-model approach that trains auxiliary models with known property ratios, extracts word-frequency statistics, and uses a meta-classifier to infer the target ratio. Experiments show that both attacks can successfully estimate dataset attributes in many cases. For instance, the shadow attack is highly effective when the model is fine-tuned in Q&A mode and the property is manifest in questions, whereas the generation attack works well in completion mode or when attributes appear in both questions and answers. The results reveal a nontrivial vulnerability that black-box LLM access can leak aggregate statistics of the fine-tuning data.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper expands on prior works on property inference and apply to LLMs.\nAuthors defined two modes and used Chat-Doctor to infer on sensitive aggregate stat which felt relevant to real world scenario. Overall, I felt the  evaluation is thorough and strong. The authors evaluate multiple scenarios and baselines, demonstrating clear cases where private proportions can be predicted. The results are well-analyzed and provides actionable insights. \n\nOne thing I wish the authors addressed is to delve into why childbirth rate performed poorly vs other properties. Also, showing whether these attacks would generalize to other domains would be valuable. \nAdditionally, I wish the authors expanded more on  potential defenses. The paper establishes the attack side but does not explore how one might mitigate these leaks.\n\nQUESTIONS:\nFigure 1. Why does \"daughter\" indicates female? Could it also be male? \nFor black box attacks on sentence completion, how many prompts and samples are generated to get a reliable estimate? How sensitive are attacks to changes in wording of prompts?\nFor the shadow attack, how large was the auxiliary dataset and how was the keyword list selected?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates property inference attacks on fine-tuning datasets of large language models (LLMs). The authors identify a gap in existing research, which has primarily focused on property inference attacks against discriminative models (e.g., image classification) and generative models (e.g., GANs), while overlooking LLMs. To address this gap, they propose a prompt-based generation attack and a shadow-model attack that leverages word frequency signals. They evaluate their approaches on the ChatDoctor dataset to show the effectiveness of their design.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper studies an interesting problem.\n2. The proposed attacks are specifically designed for LLMs and provide tailored property inference tools.\n3. The experimental results demonstrate the effectiveness of the proposed attacks and provide insightful observations.\n\nWeaknesses\n1. The gray-box setting assumes that the attacker has knowledge of the pre-trained model, fine-tuning method, instruction template, and target dataset size. This seems unrealistic for real-world scenarios where attackers typically only have API access without detailed training information. The assumption of knowing the exact dataset size is particularly strong. The authors may justify the assumptions.\n\n2. The paper lacks clear explanations of key technical details. The paper does not clearly explain how prompts are constructed for the generation-based attack, and why word frequency was chosen as the feature for the shadow-model attack. Additionally, it is unclear why existing shadow-model attacks cannot be directly applied to this setting.\n\n3. The experimental scope is limited. The evaluation is limited to a single dataset (ChatDoctor) and only two property types (gender and diagnosis). The authors may consider additional datasets and property types.\n\n4. The proposed methods target only fine-tuning datasets, potentially overlooking the more significant threat of attacking pre-training datasets, which may contain larger volumes of sensitive information.\n\nQUESTIONS:\n1. Could the authors provide stronger justification for the gray-box assumptions?\n\n2. Could the authors provide more detailed explanations of the core attack mechanisms?  Specifically, how are prompts constructed for the generation-based attack? What is the rationale for using word frequency features in the shadow-model attack, and how does this choice compare to alternative feature representations?\n\n3. Could the authors extend the evaluation to include additional datasets beyond ChatDoctor and more diverse property types?\n\n4. Can the proposed methods be adapted to infer properties of pre-training datasets, rather than just fine-tuning datasets?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}