DocMIA: Document-Level Membership Inference Attacks against DocVQA Models
Authors: Khanh Nguyen
Raouf Kerkouche
Mario Fritz
Dimosthenis Karatzas
Venue: ICLR 2025 Poster (2025)
Keywords: Membership Inference Attacks
Document-based VQA
Multi-modal Models
Privacy
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Document Visual Question Answering (DocVQA) has introduced a new paradigm for end-to-end document understanding, and quickly became one of the standard benchmarks for multimodal LLMs. Automating document processing workflows, driven by DocVQA models, presents significant potential for many business sectors. However, documents tend to contain highly sensitive information, raising concerns about privacy risks associated with training such DocVQA models. One significant privacy vulnerability, exploited by the membership inference attack, is the possibility for an adversary to determine if a particular record was part of the model's training data. In this paper, we introduce two novel membership inference attacks tailored specifically to DocVQA models. These attacks are designed for two different adversarial scenarios: a white-box setting, where the attacker has full access to the model architecture and parameters, and a black-box setting, where only the model's outputs are available. Notably, our attacks assume the adversary lacks access to auxiliary datasets, which is more realistic in practice but also more challenging. Our unsupervised methods outperform existing state-of-the-art membership inference attacks across a variety of DocVQA models and datasets, demonstrating their effectiveness and highlighting the privacy risks in this domain.

Published as a conference paper at ICLR 2025
DOCMIA: DOCUMENT-LEVEL MEMBERSHIP INFER-
ENCE ATTACKS AGAINST DOCVQA MODELS
Khanh Nguyen' Raouf Kerkouche Mario Fritz²
Dimosthenis Karatzas
:Computer Vision Center, Universitat Autònoma de Barcelona "CISPA Helmholtz Center for Information Security (knguyen, dimos (ooos))ovcuuub... waw.ea............ es (raouf. kerkouche, fritz) fritt)0ccistt pea.z//ccppp..... spa.de
ABSTRACT
Document Visual Question Answering (DocVQA) has introduced new paradigm for end-to-end document understanding, and quickly became one of the standard benchmarks for multimodal LLMs. Automating document processing workflows, driven by VoA models, presents significant potential for many business see- tors. However, documents tend to contain highly sensitive information, raising concerns about privacy risks associated with training such VQA models. One significant privacy vulnerability, exploited by the membership inference attack, is the possibility for an adversary to determine if a particular record was part of the model's training data. In this paper, we introduce two novel membership inference attacks tailored specifically to DocVQA models. These attacks are designed for two different adversarial scenarios: white-box setting. where the attacker has full access to the model architecture and parameters, and a black-box setting, where only the model's outputs are available. Notably, our attacks assume the adver- sary lacks access to auxiliary datasets, which is more realistic in practice but also more challenging. Our unsupervised methods outperform existing state-of-the-art membership inference attacks across variety of DocVQA models and datasets, demonstrating their effectiveness and highlighting the privacy risks in this domain.
INTRODUCTION
Automated document processing fuels a significant number of operations daily, ranging from fintech and insurance procedures to interactions with public administration and personal record keeping. Up until a few years ago, document processing services relied on template- information extraction models, which were created ad-hoe for each client. Although these approaches allowed for good control of client data and could be extended to new documents with few examples, they were limited in scalability and difficult to maintain. Consequently, the introduction of Document Visual Question Answering (Qo) (Mathew et al.) 2020)) in 2019 has resulted in a paradigm shift in document processing services, enabling end-to-end generic solutions to be applied in this domain. DoA leverages multi-modal large language models to streamline business workflows and pro- vide clients with novel ways to interact with the document processing pipeline.
However, as cloud-based DoeVQA solutions become more prevalent, significant privacy risks emerge, particularly concerning the potential leakage of sensitive information through model vul- nerabilities. Indeed, during the training of a Doc model, each document can have several associated question-answer pairs, with each pair considered unique data point. As a result, single document can appear multiple times, which significantly raises the risks associated with pri- vacy vulnerabilities. This repeated exposure enhances the likelihood of the model memorizing spe- cific details, thereby increasing the potential for data leakage during privacy attacks. Furthermore, scanned document images often have high resolutions necessary for posterior analysis, but need to be rescaled for processing by image encoders, potentially rendering content unreadable. To mitigate this issue, many VQA models (Huang et al. 2022 Tang et al. 2023) utilize a dual representation
Thee corresponding author.

---

Published as a conference paper at ICLR 2025
Trakkin Data
Orene medern wi the dive?)
Business Invoke
Audillory
Soo
query
DocVQA Model
250215 Maracha Thompsats...
aggregate
IN
Pre-trained Model
Which Who is cestomer? C. this contidentaa??
coumer?
Deployment
Figure 1:
The General Scheme of Document-level Membership Inference Attacks. Training: A DocVQA
model is trained datasct of documents, each associated with multiple questions'answers. Deployment: An adversary exploits this structure by querying the model with questions related a target document. By aggregating the model's responses, the adversary can infer the membership of the document in the training set.
of the document, comprising both a reduced-scale image and OCR-recognized text. This approach introduces further challenges, as sensitive information may leak through multiple modalities.
Membership inference attacks (MIAs) are among the most prominent techniques for assessing pri- vacy vulnerabilities in machine learning models. These attacks enable an adversary to determine whether specific data point is included in the training dataset. However, there is limited research on membership inference risks in the context of multi-modal models. Among the few studies, Ko al. (2023); Hu et al. (2022b) utilize powerful pre-trained models on large datasets to construct an aligned embedding space for the two modalities- and text-allowing for the inference of membership information. Unfortunately, the reliance on these pre-trained models poses challenges for document-bee based tasks, particularly in DocVQA scenarios, where an alignment model capable of aligning the (document, question) as input and the answer as output is currently unavailable. Recently, Tito et al.] (2024) introduced provider-level MIAs against DocVQA models aimed at determining whether a provider (group) that may supply multiple invoice documents is part of the training set. In contrast, specifically targeting the inference of whether a single document is included in the training dataset. Current MIA solutions that exploit standard features such as output logits, probabilities, difficult to adapt to the DocVQA context, where outputs are generated in an auto-regressive manner. Additionally, legal constraints surrounding copyright and private information complicate central- ized model training, making it challenging to create auxiliary datasets that capture the variability and richness of weal-world data, As a result, shadow training of proxy models becomes infeasible.
research focuses membership information at finer granularity,
loss
In this work, we take a structured approach to privacy testing for DocVQA models. We design novel Document-level Membership Inference Attack (DocMIA) that deals with the multiple oc- currences of the same document in the training set, as demonstrated in Figure [ To address the challenge of extracting typical metrics (e.g. logit-based) from auto-regressive outputs, we propose a new method based on model optimisation for individual samples that generates discriminative fea- tures for DocMIA. We design attacks both for white-box and black-box settings without requiring auxiliary datasets. In the black-box setting, we propose an alternative knowledge transfer mecha- nism from the attacked model to a proxy. Evaluating our attacks on three multi-modal VOA models and two datasets, achieve state-of-thee the-the-art performance against multiple basclines.
To summarize, we make the following contributions:
1. We present DocMIA, the first Document-level Membership Inference Attacks specifically targeting multi-modal models for VQA.
2. We introduce two novel auxiliary data-free attacks for both white-box and black-box set- tings, leveraging novel discriminative metrics for DocMIA.
3. We explore three distinct approaches to quantify these metrics: vanilla layer fine-tuning (FL.,, fine-tuning layer with LoRA (Hu et 202T) (FLLORA), and image gradients (IG).
4. Our attacks evaluated on two Doc VQA datasets across three different models, outperform existing -fate-of-the-eee membership inference attacks as well as baseline attacks.
Cooee available http:////ggghubb...c///aaaaagrg/ 606/11 a_docvqa
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Membership Inference Attack. Membership inference attacks have been extensively explored in various applications to highlight privacy vulnerabilities in deep neural networks or to audit model privacy (Shokri et al. 2017). These attacks are categorized into two types: white-box and block-box settings. In white-box settings, the adversary has full access to the target model's internal param- eters and computations (Carlini et al.) 2022 Yeom et al., 2018 Nasr et al. 2019 Rezaei & Liu 2021 Sablayrolles al., 2019 Li & Zhang 2021), enabling the use of informative features like loss values, logits, and gradient norms. Conversely, in black-box settings, the adversary is limited to the
model's outputs, such as predicted labels or confidence scores (Choquette- CeequeetteCChoo et al. al.] | [Shokri] et al 2017 [aalem et al., 2018 Sablayrollss et al., 2019 Song & Mittal M201 Hui et al. 20.12 D. The literature indicates that white-box attacks tend to more effective due to the availability of richer features (Song et al. 2019] Nasr et al. 2019). In this paper, we propose tailored attacks for both settings, considering a more challenging scenario where the adversary lacks an auxiliary dataset -which is used to train shadow models that mimic the behavior of the target model and are subse- quently exploited to enhance attack performance- and is restricted to a limited number of queries. Regarding -raseent-based membership inference attacks, research on using gradients as features has been limited. Nasr et al. (2019) leveraged the L2-norm of gradients with respect to model weights for membership inference. Rezaai & Liu (2021) suggested using the distance to the decision bound- ary as a metric but found it ineffective for this purpose. In contrast, we introduce novel strategies called FL, FLLoRA, and IG, demonstrating that the L2-norm of the cumulative gradient-comppln using these and Li & Zhang (202T) also explored distance metrics, but from input points for membership infer- ence in image classification tasks, their approaches lack scalability and applicability in our context, which involves larger-scale models with a wider vocabulary of tokens.
-providdesss a robust signal for membership inference. While Maini et al. (202T)
Membership Inference Attack Against Multi-modal Models. Research works into the privacy vulnerabilittiss of multi-modal models is still in its early stages. Recently. Tito et al. (2024); [into] jet al. (024) proposed reconstruction attacks that exploit VOA model memorization to recover hidden values in documents. They black out specific target values in documents and query the model with questions about the modified documents. Since the model memorizes training data, it often reconstructs the hidden target values. Tito et al. (2024) also introduced membership attack against Doc models to infer whether a document provider, with multiple documents, is included in the training dataset. However, far we know, no research has yet explored membership inference attacks at document-level granularity. Additionally, Ko et al. (2023); Hu et al. (2022b) leverage powerful pre-trained models on large datasets create an aligned embedding space for the two modalities to infer membership. Unfortunately, the reliance on these pre-trained models introduces difficulties for document- based tasks, especially Voc,QA, where an appropriate alignment model for aligning (document, question) inputs to corresponding answers is not yet available. Furthermore, the success of both attacks hinges on the availability of auxiliary datasets leveraged by the adversary. which are key to executing the attack effectively. In this paper, we present two membership inference attacks specifically tailored to tackle the unique characteristics of VQA models.
3 BACKGROUND
3.1 DOCUMENT-BASED VISUAL QUESTION ANSWERING
DQA is a multi-modal task where natural language questions are posed based on the content of document images. Notably, it establishes unified query- framework applicable across various document understanding tasks, such as document classification and information extraction.
Formally, the DoA task is defined as follows: given a question-answer pair (g, a) related to document image x, the method F must generate an answer à = F(x, q) such that à closely matches the correct answer a. More concretely, given D. = {(I,, 9: at) } as a set of valid training examples, a model F. parameterized by 0, is trained to maximize the conditional log-likelihood:
[()) log po (0s
3

---

Published as a conference paper at ICLR 2025
Standard metries for Doc include Aceuracy (ACC) and Normalized Levenshtein Similarity (NLS) [Biten et al. 2019). which measure the similarity between the predicted and correct answer:
ACC = la=a! NLS - NL(ā, (á, a) < 0.5,
(2)
NL 2 0.5
where NL(: denotes the normalized Levenshtein distance.
In the following sections, for clarity, we often omit the data example index i from the notation.
3.2 DOCUMENT-LEVEL MEMBERSHIP INFERENCE ATTACK
Membership inference attacks (Shokri et al. 2017 exploit privacy vulnerabilities to determine if a specific data point was included in the training set of machine learning model. We extend this definition to the Document-level MIA, which is particularly suited in the DocVQA context.
Given access to trained DocVQA model F and document drawn from its data distribution Ad [== related to the information in the
D, along with a set of question-answer pairs Q = {(gs, 0z) { document, adversary A designs decision rule (ar F) to classify the membership status of æ, aiming for (I, F) = 1 if x is member of the training set, otherwise non-member. It important note that the adversary is focused solely on the membership of the document x, rather than the entire DocVQA data point (I, q, a), which is typically the target of prior MI attacks. Moreover, since a single document is associated with many question-answw pairs, this allows the adversary to query the same document using multiple questions for various pieces of information.
4 DOCMIA AGAINST DOCVQA MODELS
In Section 4.1
elaborate the threat model relevant to DocMIA two scenarios: white-box and black-box access. We first explain our intuition behind our optimization-base attacks in the white-box setting (Section 4.2). then adapt this approach
black-box attacks (Section 4.3
4.1 THREAT MODEL
DocMIA can be either a useful or harmful tool in various real-world scenarios. On the positive side, DocMIA can act as privacy auditing tool. For instance, in legal document processing, law firms may use these attacks to evaluate whether proprietary or confidential documents, such as contracts or court filings, were included in model training. thereby identifying potential privacy risks. Con- versely, DocMIA can be maliciously leveraged. As an example, a business competitor could exploit these attacks data, exposing confidential business relationships and leading to risks such as supplier poaching.
invoice-proer processing system infer the presence of specific invoices in the training
In both scenarios, assume that the adversary aims to infer membership information for set of documents, determining whether each document is included in the training dataset. These docu- ments may may not be part of the target model's training data. Crucially, further assume the adversary lacks access to an auxiliary data Daux that reflects the characteristics of these documents. This assumption is realistic, as obtaining real-world documents scale is often prohibitively difficult due to their confidential nature and regulatory restrictions. Consequently, this negates the applica- tion of MI attack techniques that require training shadow models (Shokri al. 2017 Carlini al.) 2022). Even if auxiliary documents were available, training numerous shadow document- based models typically designed with large number of parameters would be prohibitively expensive.
Based on the previous examples, we refer to the owner of the document model as the trainer and the law firms or competitors as the adversary. Given the document distribution D, the trainer trains a document-based model F, with private access to D2 ~ D, following training algorithm T. that defines the model architecture, optimization process, and related details. The adversary owns the set of sensitive documents Diest D, where D; 7 Diest * 0, D Dusal = Nestt but does not know which documents are in D,. Given a document æ E Drest with a set of related queries Q { a;) H the adversary's goal is determine whether which specify the adversarial knowledge about the model F, and its data distribution D:
D, or x: & Dr. We formulate two attack settings,
White-box Setting. In this scenario, the adversary has full access to the internal workings of the target model, including the model's architecture, weights, gradients from any further training and other internal details. However, the adversary does not have access the training algorithm T.
4

---

Published as a conference paper at ICLR 2025
Black-box Setting. Here, the adversary can only interact with the target model through API, which only returns a prediction for each question q on æ. addition, the adversary is constrained by limited number of queries. As in the white-box setting, the ad has information about T. This setting reflects the most challenging case (Nasr et al. 2019; Song et al. 2019).
We assume the adversary has full knowledge of the DocVQA task to train the model, including the training objective, document type and exact training questions. This assumption is reasonable, as task-level information such document type, is often publicly available guide users, making it accessible to adversaries. The assumption of the exact question knowledge is also plausible, as an adversary can approximate questions based on the document type. Further discussion of this assumption and experiments in the setting without exact questions are provided in Appendix F2
4.2 WHITE-BOX DOCMIA
In the white-box setting, where the adversary has access to the trained model, shadow training is impractical due to the lack of auxiliary data and high computational cost. To this end, our strategy develop unsupervised metric-based attacks (Hu al. 2022a). For each document, we extract a set of features from individual question-answer pairs and aggregate them across all pairs. We then cluster the resulting feature vectors distinguish member from non-member documents. A key challenge is to design discriminative features, as standard metrics (e.g., logit and loss) may be ineffecctive this setting (Section ( To address this, we propose new features that enhance the informativeness of our membership inference vectors.
4.2.1 OPTIMIZATION--ASEI DISCRIMINATIVE FEATURES
In this section, we introduce two novel discriminative membership features derived from an opti- mization process for attacks against Doc models.
Intuition. Since VQA models are typically trained on multiple question-answer pairs per document, the model parameters likely converge to minimize the average distance to the ground-truth an- swers after training. As result, fine-tuning the model on one question-answe pair through an iterative process is necessary to ex- tract more reliable membership signals. More importantly, this op- timization on training documents may converge faster than to non- training documents, due to the lower generalization error. Figure illustrates our reasoning.
O
Palaglann (2)
Pa(ai|2inn (1)
We provide a formal definition of the distance feature.
x in of the training set
Definition 4.1 (Optimization-based Distance Feature). Given model F parameterized by 8, let the model be initialized with 6g- After undergoing gradient-based optimization process O, the pa- rameters converge to H according to a specified training objec- tive C. The distance feature is then defined the L2-norm of the change in parameters:
Paint Winat-4h
Palogl qa.
A 0*) 1100 0-1
Palogltoud 92.
This feature measures the difference between the initial parameters Oo and the converged parameters 0", as an approximation of the optimization trajectory toward the optimal solution.
x is out of the training set
Figure 2: Visualization of our fine-tuning strategy in the pa- rameters space. Each contour plot represents the optimization landscape w.r.t cach pair (at, q.) from document In general, the average A computed on a mem- ber document Tin is smaller than non-member document Zont-
Specifically, we fine-tune the target DocVQA model on an individ- ual documen/qqessiionaansww
pair and compute the distance re- quired to reach the optimal answer. A small average distance indi- cates the document likely part of the training set, while larger distance suggests non-training document. In addition, the number of optimization steps serves as an orthogonal feature that reflects the efficiency of the optimization process. With optimal learning rate and good initialization provided from the target model, optimization for training documents typically converges in fewer steps compared to non-training documents. Consequently, we include both the distance and the number optimization steps in our feature set for white-box attacks.
5

---

Published as a conference paper at ICLR 2025
Algorithm DocMIA Assignment
1: Input: model Fe,, document x E Dust question-answer pairs ((g,, a.) utility 24 aggregation +
Hyperparameters: optimizer OPT, optimization steps S. learning rale a, threshold т.
3: for to M do
4: Set O.requires.grer = True // Change 0 to: Oz or LORA(Or.) and Freeze 0.
5: Initialize: 8, = 0, u, (:: 4 + 0, 0o + 8:
while s, do
7: u; + u; ( 9.).
8: if (C(0) 1) r then break; // Early stopping end if
9:
10:
0 Oa,(a, vo(((())
11:
I +- [(0), 8: + 8: +1
12:
end while
13:
A. < 1100 0| // Compute distance metric
14:
end for
15: As ((A
M)) 8M (($ uss ((u.=1.....
Aggregating over M questions
16:
Output: P. = AMs 835, uar)
// Assign membership feature vector
4.2.2 METHODOLOGY
We now formally present our attack strategy, assuming white-box access to the target model F;- For any document x Drest and a of question-answer pairs Q, the goal is assign features descriptor Fz. This is achieved by first extracting set of features through the optimization process O on a single question-answe answer pair. These features are aggregated across multiple questions then concatenated to construct Fz. Repeating this process over Dies, we apply an unsupervised clustering algorithm to differentiate member documents from non- members based on their features descriptors.
Following our intuition, for each question-anse ansoerr pair (g, a). we fine-tune the target model parame- ter O, via gradient descent to maximize po(a) x q) (Eq. D. Starting from 0,, optimization proceeds with learning rate a, querying the model at each step using & and evaluating prediction à via utility function Ut either ACC or NLS (Eq, 2 The process stops when no further improvements is ob- served, governed by threshold number of steps s, and aggregate the utility evolution to obtain the overall Doc VQA score u. These features serve membership signals for the current (q, a) pair in relation the target document x.
max S steps. We then evaluate the distance A (Eq. 3.. record the
Since each document is associated with a varying number of question-answer pairs M, we aggregate features across all pairs using क optionally employing multiple aggregation functions enrich the feature set. The aggregated features are then normalized onto consistent scale and concatenated to form the document descriptor Fz. The assignment algorithm is outlined in Algorithm [ Finally, we cluster descriptors from Des, identifying the cluster with the larger A as non- member documents.
4.2.3 IMPROVING EFFICIENCY
Fine-tuning F. on a single (q, a) enables membership inference but computationally expensive due to model size and data pre-processing overhead. To improve the attack efficiency, we introduce three variants of the method, as illustrated in Figure 3
Optimize One Layer (FL). Instead of updating all parameters, we optimize single layer, assuming its gradients provide sufficient signal for membership classification. In this variant, we select one specific layer /, to optimize, keeping the other parameters fixed. We ablate the choice of T. for this method in Appendix C In addition, we propose FIL.RA, a variant using LoRA (Hu et al. 2021), where the LoRA parameters
initialized with Kaiming initialization (He et al. 2015). 5. From Algorithm we replace 9 to 0r, or the LORA parameters of L. denoted as LORA respectively.
Optimize the Document Image (IG). This variant shifts the focus to the input space by directly optimizes the pixel values of the document image x. The intuition remains the same: training doc- uments require less self-tuning, enabling the model to converge faster than non-tran tra-traanings. However, this assumes the target model allows differentiation of the document image through its architecture. Accordingly, we replace with from Algorithm T while freezing the target model parameters 0.

---

Published as a conference paper at ICLR 2025
C(0;a)
C(0: a)
C(w, Fo 2)}
predicted
predicted answer
answer
answwer
Layer (LoRA)
White-box + Model
White-box T Model
Proxy Model
Black-box Model
quastion
question
document
documant
docament
Optimize the Document
(b) Optimize One Layer
(c) Distill the Black-box
Figure 3: Variants of our proposed DocMIA. Left: (a) (b) illustrate three white-bos attack strategies: optimiz- ing either the Document Image single Layer (LoRA). Dashed arrows indicate back-propagated gradients. Right: We distill the black-box model into a proxy. which is then attacked using the white-box strategies.
These variants reduce computational costs while maintaining attack performance, providing more practical options when the size of Diest increases.
4.3 BLACK-BOX DOCMIA
In the black-box box setting, the attack model's access is restricted to Dies and the predicted answers. To address this, propose distillation- -based attack that transfers knowledge about the private data Dr from the black-box model Fe to a proxy model Fp. parameterized by w. With full control over Fpp we can apply white-box attack strategies without modification.
First, the black-box model generates labels for each question in Diest creating a query dataset Nia
Dquery- = (x.,, 2ey where Q; = Fo, (#i, ) The proxy model Fp is then trained on this dataset to maximize the likelihood of the predicted answer (Fo, (I; Qj [x;) qz)- In essence, the goal is replicate the black-box model's predictions and transfer its output structure the proxy model, thereby transferring the embedded membership features and validating the white-box attack assumptions. Figure ((c) illustrates the scheme of our proposed attack.
Since our focus is on the document domain, we initialize Fp using a publicly available checkpoint Wpt. pre-trained on an unlabeled document dataset Dpt. which inaccessible and assumed to be disjoint from the private dataset D,. This initialization equips the proxy model with document understanding capabilities without prior knowledge of D,- As result, it enables the proxy F, to better mimic the prediction behavior and internal dynamics of the black-box F, after fine-tuning.
important to note that the adversary lacks information the black-box training algorithm T, meaning there is no advantage regarding model architecture training details when constructing the proxy model. Thus, the choice of the proxy model, optimizer, lcarning rate, etc., is independent of the target model. However, empirically shown in Section 6. while using proxy model with the same architecture as the black-box model can offer benefits, our attack strategies remain effective with different architectures. This demonstrates the robustness of our approach, which does not rely on specific model classes or requiring detailed knowledge of the black-box model.
EXPERIMENTAL SETUP
5.1 TARGET DATASET AND MODEL
Target Dataset. We study two established DocVQA datasets in the literature for our analysis: DocVQA (DVQA) (Mathew et al., (02T) and PFL-DocVQA (PFL) (Tito et al. 2024). Both datasets designed for extractive DocVQA task, where the answer is explicitly found within the document image. Each document in these datasets is accompanied by varying number of questions.
Target Model. We consider three state-of-the- art models which designed for document under- standing tasks: (1) Visual T5 (VT5) (Tito et al. 2024) (250M parameters) follows the traditional design by utilizing OCR module to facilitate the reasoning process. It leverages the T5 model pre-
7

---

Published as a conference paper at ICLR 2025
Model
SCoRn-TA
ScoRi-HA
ScoRt-UAA
Loss-TA
GRADENN-UA
SconeLLoss-UAi
ACC
F1
ACC
Fl
ACC
F1
ACC
F1
ACC
F1
ACC
F1
PPL
VT5
62.33
64.13
61.00s.
68.80..
00.07c.0
60:675.
62.81
60.67e.
00.67aa
60.67Ga
60.67aa
Donut
73.33
75.14
75.77e.
71.17oa7
71.33
78.99
70.076.s
03.56
70.83c.
69.076.5
VTS
75,67
75.75
72 1760
76.18g.
75. 17
75. 13
73.67
77.99
71.17us
67.54lu
75.50u
76.02ax
Donut
79.67
79.53
75 75
79.57
80.50
81.10u
51.83
B3-46
77.176e
75.42oo
80.50cc
81.10ka
Pix2Stract-8
67.97
68.17m
71.36a
63.13.
64.63
08.326.
69.00ca
67.48
Table I: Results from Baseline Attacks. Gray color indicate attacks conducted the black-box setting. All results are reported based on five random seeds for KMEANS. The methods with the best average performance across the two metries are highlighted in bold. See Appendix
E for the complete results.
trained on the C4 corpus (Raffel et al. 2020), along with a ViT backbone pre-trained on document data (Li et al., 2022). (2) Donut (Kim et al. 2022) (201M parameters) is one of the first end-to- end DocVQ* models capable of achieving competitive performance without relying
OCR. It is
pre-trained on a large collection of private synthetic documents. (3) Pix2Struet (Lee et al. 2023) is
another OCR-free document model with two versions: Base (282M) and Large (1.3B parameters). This model is pre-trained to perform semantic parsing on an 80M subset of the C4 corpus.
For the PFL-DocVQA dataset, consider two targets: VT5, using the public checkpoint provided by the a and Donut, which we successfully trained to achieve strong performance following the training procedure from the authors. For the Doc dataset, we attack four targets: VT5, Donut, and Pix2Struct (Base and Large), all with publicly available checkpoints from Hugging- Wolf ( et al. 2020). In the black-box setting, we use VTS and Donut as proxy models. To train the proxy models on the query set D Qauery. we initialize them with their public pre-trained check- points the same checkpoints used to fine-tune the target models on the respective target datasets, as outlined in the original papers. For more details on datasets and models, see Appendix A and
5.2 IMPLEMENTATION
We tune the hyperparameters in the optimization process to ensure our attacks are effective against each target model in the white- box setting, then apply the best set to black-box attacks. Assuming no access the training algorithm T, we use Adam (Kingma 2014) as the optimizer OPT across all attack experiments. We explore the impact of learning rate a, the selected layer L, and we carefully tune the values of threshold т the ablation study (Appendix C). Subsequently, the optimal set of hyperparameters for each model is then applied in all black-box experiments. For aggregation 4, we consider 4 aggregation functions {AVG; MIN; MAX; MED} for each feature, denoted as Pan- Throughout experiments, employ KMEANS the clustering algorithm. See Appendix D for more implementation details.
5.3 EVALUATION METRIC
From the official splits of each target dataset, we sample 300 member documents from the training set and 300 non-member documents from the test set, yielding Niest = 600 test documents. We evaluate attack performance using Balanced Accuracy and Fl score in the balanced setting, in prior works (Salem et al.) 2018 Watson et al.) 2022 Ye et al. 2022). In addition, we report True Positive Rate (TPR) 1% and 3% False Positive Rate (FPR), as in recent MIA literature (Carlini ct al., 2022). For unsupervised attacks using KMEANS, the membership score of each document is computed as the Fuclidean distance between its feature vector and the member cluster centroid.
5.4 BASELINE
In the black-box setting, we evaluate three MI attacks that only requires the predicted answer to determine membership: Score-TTheshher Attack (SCORE-TA), Unsupervised Score-based Attack with AVG (SCORE- UA) (Tito et al. 2024) and T (SCORE-UA..))
For the white-box setting, where loss or gradient is accessible, we evaluate three base- lines: Loss-Threshold Attack (Loss-TA) (Yeom et al. 2018), Unsupervised Score+Loss Attack (SCOREL. and Unsupervised Gradient Attack AAAADINNTUUANNaSS et al. 2019). In
https::/huugiinggaacs co/models
8

---

Published as a conference paper at ICLR 2025
DocVQA
DVQA
PFL
VT3
Donut
Pix2StncctB
Donut
Loss-TA
14.00
7.67
5,33
3.00
14.67
GRADIENT- UA
9.33
6.00
5.00
3.00
8.53
55
SCORELOSS- UAai
4.67
8.67
6.67
4,00
6.33
PFLOGCVOA
Min KS Prob!
10.67
1.33
5.33
0.00
Min-K9++
7.00
9.33
10.33
8.00
2,00
FL. FLLORA
5.67
10.67
11.00
8.67
7.00
11.33
5.33
6.33
3.33
10.00
5.67
8.00
10.33
11.00
Table 2: White-box Setting: TPR at 3% FPR. Com-
Vib
parison across all white-box methods, with the hest- performing method for each metric highlighted in bold. Figure 4: White-box Setting: Our proposed at- i indicates baselines with grey-box access. We refer the tacks consistently achieve high performance, gen- readers to Appendix B for the complete results. erally outperforming the considered basclines.
addition, we include two grey-box attacks designed for LLMs: Min-K% Prob (Shi et al. 2023) and Min-K%+++ (Zhang et al. 2024), assuming access to token-level probabilities of the generated an- swer compute the membership score of each document. See Appendix detailed descriptions.
6 EVALUATION
6.1 WHITE-BOX SETTING
Baseline Performance Evaluation. (right]) shows the performance of baseline attacks in the white-box setting. Loss-TA, TA, akin to the thresholding loss attack in (Yeom et al. 2018), performs poorly complex Doc models, achieving under 60% accuracy for most targets. In contrast, SCOREL-SS--AA. which combines utility scores and loss features, achieves stronger results: 81 % Fl on Donut, 75% on VT5, and 69% on Pix2Struct on VQA dataset. However, it underperforms LOss-TA on FFL-DocVQA, with a 3% drop in Accuracy and 8% in F1, likely due to high loss variance in this dataset. GRADIENT-UA UA, which incorporates one-step gradient information, matches the performance of score-based attacks, suggesting that the gradient serves as a useful signal for membership inference. However, none of the baselines generalizes well across all target models.
Our Proposed Attacks outperform the Baselines. We evaluate our attacks-Fi. FL,LRR, and IG -n the white-box setting against different targets. As shown in Figure 4 our methods consistently achieve high performance, indicating that optimization-basse features generalize well across various models. Compared to all baselines, our attacks achieve either the best or near-best performance on both target datasets, with notable Fl scores of 72% against VT5 and Pix2Struct, and 82.5% against Donut. Against GRADIENT-US UA, our proposed features yield up to a 10% improvement in F1 on Donut, indicating that single-step step gradients are insufficient for reliable membership inference.
From Table 2 our attacks consistently perform well in the low-FPR regime, often surpassing or matching the strongest baselines. For instance, FL. achieves TPR of 8.67% 3% FPR against VT5 on PFI,, despite minimal overfitting, and a TPR of 1 1.00% against Pix2Struct-B on Doc Addi- tionally, our methods outperform both Min-K% Prob (Shi et al. 2023) and Min-K%++ (Zhang al.) 2024) across target models, underscoring their effectiveness, particularly in DocMIA setting. These results highlight the privacy risks posed by cidimization-basee features in membership inference.
Why are Our Attacks more effective? We evaluate our optimization-basel features against stan- dard metrics from prior MI attacks such as loss and single-step gradient. The loss-based attack LOSS-TA (Yeom et al. 2018) assumes that member documents exhibit lower loss values than non- members after training the target model Fr. While this exploits the generalization gap, is overly simplistic for large-scale models that are trained with techniques to minimize overfitting. Conse- quently, the strong generalization capability of DocVQA models often reduces the sensitivity of the loss membership indicator. In contrast, our attacks leverage the optimization process with respect to the model parameters, conditioned on each dnesttionanswer pair. We hypothesize that the distance resulting from parameter updates (Eq. 3) is smaller for member documents than for non-members, as depicted in F This fine-grane iin-ggrinndd signal, which captures the model's 's internal response to optimization, provides more discriminative membership feature.

---

Published as a conference paper at ICLR 2025
Nodi
DV-AA
P9L
lack
Sccrua TA
M.33
13/M1
5/41
9:80
1.00
2.7
Dewe
73 114
16 14
16 18
26.5 (-455)
Scoan UAA
9.33
2.67
VTE
13
23,00
16.87
2.00
Priliienn
TI 11
68 68
Ril v-230
FLLaKA
10
8:11
7.00
7.67
7:38
1.53
6.67
DPON TXY
Besy Nast
XTA
R0aaal1100 25
Moad 170
6 3.0
Doca
62 Paul 10
69 thui 1:50
That 11.31
Donat
FULORA
6:33
5.00
11/M
5.00
6.38
935
6.33
8.00
539
6.33
533
4:31
Table 4: Black-box Setting: TPR at 3% FPR using Donut and VT5 as proxy models. Comparison across all black-hox methods, with the hest-performing method points for the black-box models are trained on the respective for each metric highlighted in bold. The datasets. Values in parentheses indicate the improvement (pos- complete results can be found in the Ap- itive/negative) compared to the best number from SCORE-UA- pendix E based baselines. Results are reported over five random seeds.
Dess
43 4.0
8088u (4822
64 -12
Table 3: Black-Box Setting: Main Results of Black-Box DoeMIA using Donut and VT5 as proxy models. The check-
As illustrated Figure X our distance feature better separates between members and non-members compared to loss-based methods (Figure 7 (top)). The t-SNE visualization (van der Maaten & Hinton 2008) from Figure 7 (bottom) further show that features derived from our attacks create more distinct clusters in high-dimensional space across all target models, outperforming loss- approaches. For further analysis on the impact of our selected features, see Appendix FF
6.2 BLACK-BOX SETTING
Baseline Performance Evaluation. Table T (left) presents the results of our black-box baseline attacks, all of which rely on the DocVQA score as the only source of information in this setting. Similar to the loss metric, the score metric is directly correlated with the generalization gap, making attacks more effective when there is a higher degree of overfitting. This trend is illustrated in Figure 9 where we observe strong MI performance, particularly for the Donut model with 75% in PFL and 79% Fl score in VQA. Meanwhile, both SCORE-UA-basee baselines (Tito et al. (024) show comparable performance, especially effective against models trained on VQA. Overall, no single method emerges as the clear winner across all target models.
Attacking via Proxy Model. Table and Table present the key results of our black-box attacks.
First, we observe a clear advantage of attacking the proxy models distilled with our proposed tech- niques. Across a wide range of black-box architectures trained on both target datasets, attacks leveraging the proxy models outperform the black-box baselines in most cases, demonstrating better MI performance. This suggests that, even without knowledge of the black-box model architecture, one chosen proxy model still effectively distills certain behaviors from the black-box models which are membershhip-nndicaive,,, enabling our attacks to infer membership with high accuracy.
When the black-box model architecture matches that of the proxy, we consistently observe improve- ments in MI performance, especially when targeting the PFL-DocVQA dataset. Among the target, Pix2Struct is the most vulnerable (both Base and Large). Both VT5 and Donut proxies gains of +3.04% in Accuracy and +4.88% in Fl score over the best baseline, even against the Pix2Struct-L. model. which exhibits strong generalization and a minimal Train- Test gap (Figure 99). Furthermore, proxy VT5 can achieve TPRs of 23. 23.00% and 16.67% against Donut and Pix2Struct-B, respectively, at 3% FPR on VQA. We also provide analysis of the proxy model in Appendix F3
These results suggest that privacy vulnerabilities can be exploited using simple distillation-basel strategies applied to the model's output space.
7 CONCLUSION
In this paper, we introduce the first document-level membership inference attacks for DoA mod- els, highlighting privacy risks in multi-modal settings. By leveraging model optimization techniques, we extract discriminative features that address challenges posed by multi-modal data, repeated doc- ument occurrences in training, and regr-ssgee outputs. This enables us to propose novel, aux- iliary data-free attacks for both white-box and black-box scenarios. Our methods, evaluated across multiple datasets and models, outperform existing membership inference baselines, emphasizing the privacy vulnerabilities in VOA models and the urgent need for stronger privacy safeguards.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
This work has been funded by the European Lighthouse on Safe and Secure AI (ELSA) from the European Union's 's Horizon Europe programme under grant agreement No 101070617. Views and opinions expressed are however those of the authors only and do necessarily reflect those of the European Union or European Commission. Neither the European Union nor the European Com- mission can be held responsible for them. Khanh Nguyen and Dimosthenis Karatzas have been supported by the Consolidated Research Group 2021 SGR 01559 from the Research and Univer- sity Department of the Catalan Government, and by project PID2023-14642666 MCIU/AEE/1O. 11101/0...0000000000000000000000 00000/551110011111100 1033 and FSE+.
funded by
ETHICS STATEMENT
Our research introduces two novel membership inference attacks on Doc models, designed to evaluate the privacy risks inherent in such systems. While our methodology exposes vulnerabilities that could potentially be exploited for malicious purposes, the primary objective of this work is to raise awareness about privacy issues Al systems, specifically the context of DocVQA models, and to encourage the development of more preeerviig technologies.
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference computer and communications security, pp. 308-318- 2016.
Borja Balle, Gilles Barthe, Marco Gaboardi, Justin Hsu, and Tetsuya Sato. Hypothesis testing interpretations and renyi differential privacy. In International Conference on Artificial Intelligence and Statistics, pp. 2496-2500 PMIR, 2020.
Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4291-4301, 2019.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Mem- bership inference attacks from first principles. In 2022 IEEE Symposium Security and Privacy (SP), pp. 1897-1914. IEEE, 2022.
Christopher A Choquette-Choo Florian Tramer, Nicholas Carlini, and Nicolas Papernot. Label-only membership inference attacks. International conference on machine learning, pp. 1964-1974. PMLR, 2021.
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends@ in Theoretical Computer Science, 9(3-4)2211-007 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
Hongsheng Hu, Zoran Saleie, Lichao Sun, Gillian Dobbie, Philip S Yu, and Xuyun Zhang. Mem- bership inference attacks on machine learning: A survey, ACM Computing Surveys (CSUR), 54 (11s):1-37, 2022a.
Pingyi Hu, Zihan Wang, Ruoxi Sun, Hu Wang, and Minhui Xue. M'I: Multi-modal Models Mem- bership Inference. Advances in Neural Information Processing Systems, 35:1867 1862118778, 2022b.
Yupan Huang. Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layoutlmv3: Pre-training for document ai with unified text and image masking. In Proceedings the 30th ACM International Conference Multimedia, pp. 4083-4091 2022.

---

Published as a conference paper at ICLR 2025
Bo Hui, Yuchen Yang. Haolin Yuan, Philippe Burlina, Neil Zhenqiang Gong, and Yinzhi Cao. Practical blind membership inference attack via differential comparisons. arv preprint arXiv:2101.01341, 2021.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint #xiv:2310.006225 2023.
Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam. Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document un- derstanding transformer. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision - ECCV 2022, pp. 498-517, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031-198155-1
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.660 2014.
Myeongseob Ko, Ming Jin, Chenguang Wang, and Ruoxi Jia. Practical membership inference at- tacks against large-scale multi-modal models: A pilot study. In Proceedings of the TEEE/CVF International Conference on Computer Vision, pp. 4871-4881, 2023.
Kenton Lee, Mandar Joshi, Tulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khan- delwal, Peter Shaw, Ming-Was Wei Chang, and Kristina Toutanova. Pix2struct: screenshot pars- ing as pretraining for visual language understanding. In Proceedings of the 40th International Conference on Machine Learning, ICML'22 23. JMLR.org, 2023.
Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, and Furu Wei. Dit: Self-supervised pre-training for document image transformer. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 3530-3539, 2022.
Zheng Li and Yang Zhang. Membership leakage in label-only exposures. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pp. 880-895, 2021.
Pratyush Maini, Mohammad Yaghini, and Nicolas Papernot. Dataset inference: Ownership resolu- tion in machine learning. 2021.
Minesh Mathew, Ruben Tito, Dimosthenis Karatzas, R Manmatha, and CV Jawahar. Document visual question answering challenge 2020. arXiv preprint arXiv:2008.0883 2020.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference applications of computer vision, pp.
2021.
Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning: Passive and active white- box inference attacks against centralized and federated learning. In 2019 IEEE symposium security and privacy (SP), pp. 739 -53.. IEEE, 2019.
Francesco Pinto, Nathalie Rauschmayr, Florian Tramèr, Philip Torr, and Federico Tombari. Extract- ing training data from document-basee VQA models. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, pp. 40813-40826. PMLR, 2024.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Shahbaz Rezaci and Xin Liu. On the difficulty of membership inference attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7892-7900, 2021.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Herve Jegou. White- box vs black-box: Bayes optimal strategies for membership inference. In Proceedings of the 36th International Conference on Machine Learning, pp. 5558-5567. PMLR, 2019.
12

---

Published as a conference paper at ICLR 2025
Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. MI-leaks: Model and data independent membership inference attacks and defenses on machine learning models. arXiv preprint arXiv:1806.01246, 2018.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. arXiv preprint 1:Xiv:2310.1678,, 2023.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
Liwei Song and Prateek Mittal. Systematic evaluation of privacy risks of machine learning models. In 30th USENIX Security Symposium (USENIX Security 21), pp.
2021.
Liwei Song, Reza Shokri, and Prateek Mittal. Privacy risks of securing machine learning models against adversarial examples. In Proceedings of the 2019 ACM SIGSAC conference on computer and communications security, pp. 241-257, 2019.
Zineng Tang, Ziyi Yang. Guoxin Wang, Yuwei Fang. Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, and Mohit Bansal. Unifying vision, text, and layout for universal document processing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
19254 19264, 2023.
Rubèn Tito, Khanh Nguyen, Marlon Tobahen, Raouf Kerkouche, Mohamed Ali Souibgui, Kangsoo Jung, Joonas Jälkö, Vincent Poulain DAndecy, Aurelie Joseph, Lei Kang, et al. Privacy-aware document visual question answering. In International Conference on Document Analysis and Recognition, pp. 199-218. Springer, 2024.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, pp. 2579 20099, 2008.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Lauren Watson, Chuan Guo, Graham Cormode, and Alexandre Sablayrolles. On the importance of difficulty calibration in membership inference attacks. In International Conference on Learning Representations, 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-ar natural language processing. Qun Liu and David Schlangen (eds.), Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 38-45, On- line, 2020. Association for Computational Linguistics.
Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. En- hanced membership inference attacks against machine learning models. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security pp. 3093-3106, 2022.
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learn- ing: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pp. 268-282. IEEE, 2018.
Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang. Hao Frank Yang, and Hai Li. Min-k%+++ Improved baseline for detecting pre-training data from large language models. arXiv preprint Kkiv:2404.029366 2024.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
A DocVQA VQA Datasets
14
Baselines
15
Ablation Study
16
D More on Attack Implementation
17
D.1 Target Model Training
17
D.2 Target Model Performance VQA
18
D.3 Computation and Runtime
19
More on Attack Results
19
More on Analysis
19
Impact of Selected Features
20
R2
Impact of the Training Questions Knowled
21
The resulting Proxy Model
22
44
Attack Performance against Minimal- Training Documents
23
Defenses
23
A DOCVQA DATASETS
DocVQA (Mathew et al.
2021 This dataset contains high-quality human annotations and is widely used as benchmark for document understanding. It comprises of real-world administrative docu- ments across diverse range of types, including letters, invoices, and financial reports.
PFL-DocVQA (Tito et al. 2024) A large-scale dataset of real business invoices, often containing privacy-sensitive information such payment amounts, numbers, and bank account details. This dataset is specifically designed for DoA tasks in federated learning and differential privacy setup, supporting different levels of privacy granularity. The dataset is accompanied by variant of MI attacks, where the goal is to infer the membership of the invoice's owner (i.e., the provider) from a of their invoices that were not used during training.
VQA
PFL-DoeVQA
Split
Num. Docs
Num. Questions
Num. Docs
Num. Questions
Train
69894
221316
10194
39463
Val
91.50
30491
1286
5349
Test
13463
43591
1287
5188
Table 5: Statistics from PFL and DocVQA dataset.
we present statistics for both the DocVQA and PFL-DocVQA datasets. Additionally, Figure shows the distribution of questions per document: (1) while small subset of documents have more than 10 questions, most contain fewer, and (2) a fraction documents have only a single question. These trends hold across both datasets.
In Table 5
14

---

Published as a conference paper at ICLR 2025
Train
Train
3000
2500
1000
300
300
14
PQuectioe Oocament
Squndieesppr Decumeet
PQueetices Docamatt
#Quactiin per Decumwet
(a) DocVQA
(b) PFL-DocVQA
Figure 5: The distribution of number per-document questions from PFL and Doc dataset.
B BASELINES
For the black-box setting, evaluate three MI attacks baselines, which only requires generated text to infer the membership of the target document:
Score-Threshold Attack (SCORE-TA assumes that training documents should achieve higher scores than non-training ones. This attack, adapted from Yeom et al. (2018), evaluates the pre- diction for each question q using the utility function U and computes the average score A document is then predicted as member ū 2 K, and non-member otherwise. The threshold x is set the average value of across Diest-
Unsupervised Score-based Attack (SCORE-UA) (Tito et al. 2024 This attack applies an unsu- pervised clustering algorithm over the set of average score from test documents in Diest documents within the cluster with higher average score are predicted as members.
Unsupervised Score-based Attack An Extension (ScORE-UAat). This attack extends SCORE-UA UA by considering multiple aggregation functions Lall to form the feature vector.
For the grey-box setting, we consider two additional baselines which assumes access to token-level probabilities of the generated answers a to compute the membership score of each document:
Min-K % Prob (Shi al. 2023) computes the average log probability of the lowest-K% answer tokens as the membership score: Min-K% Prob = Min [a] La, K(i) log pogp((a/<<i) Intu- itively, training documents are less likely to contain low-probability answer tokens, resulting in higher scores.
2024) also averages scores from the lowest-K% probability tokens but assumes that tokens in the predicted answers for training documents have high probabilities or often form the mode of the conditional distribution. Thus, for each token, the score is computed
Min-K% +++ (Zhang et al.
Min-K%*+4 %+++(2i ax) log paalll with Haçe and Oaci are the expectation and standard daze
deviation p(a, a<)) respectively.
We adapt these baselines to DocMIA by using AVG aggregation function to combine scores across question-answer pairs within document. We evaluated K [0.6, 0.7, 0.8, 0.9, 1.0), which correspond to corresponds to 60% 100% the length of the answer and reported the best result.
In the white-box setting, where loss information is available, we consider three additional baselines:
coss-Threshold Attack (LOSs-TA. TA) (Yeom et al., 2018) Similar to SCORE-TA, this attack com- 1 M (F, q:)). A document is predicted as a member if S k. and putes the average loss 37
otherwise non non-member, where x is selected as the average value of across Djest-
Unsupervised One-step Gradient Attack (GRADIENT- UA) Inspired from Nasr al. 2019), this attack utilizes the average norm of the gradient of the loss VoL from a single optimization step. also incorporates the average
u. both aggregated with & the features to perform clustering.
Unsupervised Score+Loss Attack (SCoRELOSS: Loss-UAAA This attack extends SCOREUA com- bining the average loss I with the average utility score ū, then aggregating with Panl-
15

---

Published as a conference paper at ICLR 2025
200
AKC
50.0
(a) Learning Rate a
(b) Threshold
Figure 6: Ablation Study Learning Rate and Threshold The best value for each model across all datasets used as the hyperparameters our black-box attacks.
Layer
VT5(FFL)
Donut(DocVQA)
Pix2Struc-B(DoccOO0
Emhedding Projection Layer
67.0
71.33
68.66
Embedding Layer Norm
65.33
76.0
64.67
Last Decoder Block FCI
68.33
78.0
68
Last Decoder Block FC2
68.17
77.33
68.83
Last Decoder Block Layer Norm
61.83
76.83
67.5
Random Decoder Block FCI
61.33
72.0
67.5
Random Decoder Block FC2
64.0
73.0
65.17
Table 6: Effect selected layer tune from each target model. Attack performances are reported in terms Accuracy.
C ABLATION STUDY
In this section, we provide a detailed analysis of the hyperparameter tuning process for DoeMIA in the white-box setting, targeting all the considered models. Given the high computational cost due to the numerous factors involved, we focus on the key parameters that may potentially affect the attack performance. Our intuition behind this tuning process that: achieving a reliable estimate of distance A requires the optimization process to converge effectively, which in turn correlates with higher attack accuracy. Thus in all our experiments, to increase the likelihood of convergence, set the maximum number of optimization steps to S 200. We fix the maximum number of questions per document M to 10.
Learning Rate We first study the effect of a, which controls the speed of the optimization process in our attacks. This threshold r is empirically set to be the average loss change observed when performing one optimization step after reaching the correct answer. Only the distance A and the number of steps are used as the features. For FL and FLLoRA attacks, we perform hyperparameter search over grid of learning rates, (0.001, 0.01, 0.1, 0.5, 1.0, 5.0, 10.0, 20.0) for the IG attacks. For FL and FLLoRA, we specifically tune the embedding projection layer, which projects the final hidden states into the vocabulary space, common design choice across all the target models considered.
{10-1 4, 0.001, 0.01, 0.1, 0.5, 1.0), and
As shown in Figure setting high learning rate can cause the optimization process to overshoot, while lower values lead to a more stable but slower convergence. We find that learning rate of a 10 consistently delivers the best attack performance across most of the settings.
The layer to tune L. We now investigate the impact of layer selection on the performance of our and FI.LORA attacks. All target models our study follow the transformer encoder-decoder architecture (Vaswani] 2017), where each component consists of stack of attention layers, and shared embedding projection layer maps the hidden states to logit vectors for prediction. Given this common structure, examine the effect of tuning similar layers across all models, with results for attack accuracy presented Table 6
Our findings reveal that layers closer to the final output exhibit higher privacy leakage in terms of MI compared to (randomly selected) intermediate layers, likely due to receiving larger gradient updates. Specifically, fine-tuning the final fully connected layer alone leads strong attack performance
16

---

Published as a conference paper at ICLR 2025
Model
OFL
OXG
S
TPL
71G
VT5
1.0
10-6
10
Donut
0.001
0.001
200
last FC layer
1.0
5.0
Pix2Struct-B
0.001
10 10-4
10-3
Table 7: Best Hyperaremeters from our tuning process with consistent performance across both PFL and Doc dataset.
while also being more efficient in terms of the number of parameters that need to be optimized. This suggests that focusing on the last layers can achieve both high privacy leakage and computational efficiency in our MI attacks.
Threshold т. With the optimizer OPT and learning rate a fixed, the threshold t emerges as the most critical hyperparameter that requires careful tuning for each attack. We experiment with wide range of values, spanning from 10-6 to 10.0, and select the optimal value based on attack performance, as demonstrated subsequent experiments. Careful selection of this threshold is crucial, as it directly influences the stability and success of the optimization process.
Figure 6(b) This optimal T is then applied consistently in all
We summarize the set of tuned hyperparameters for our approach in Table
D MORE ON ATTACK IMPLEMENTATION
D.1 TARGET MODEL TRAINING
For all target models, whenever feasible, we utilize the public checkpoint fine-tuned the consid- ered private dataset from Hugging Face library and adhere to the data processing guidelines, such as document resolution, as recommended by the authors. We deliberately opt for public checkpoints for two reasons: (1) to make it consistent to further research in privacy attacks that use the same trained models, and (2) to minimizing the biases in model training that affect the final results, given the complexity of the original training process and our limited resources. Table & summarizes the details of the process from which public checkpoints for the target models considered in this work obtained. This includes the datasets the models were pre- ireitrained on, before by fine-tuning target Doc VQA datasets, along with the corresponding download URLs for these checkpoints.
Model
Tiak
Checkpoine
Phetrinn
Prnetune
Premai
Pinetune
VTS
25004
DecVQA
C4 CDPP
DocVDA
PRL
brps www.ehehhhaaaaaaaeo www.eaakkkaaaaiiiiiiiao..a. di.cultbe2
Donut
20004
DeeVQA
CDMP 1IM 0.5 systhsized Docs
DaeVQA
PR.
mnnrrcinneeseeeen hrae fisctunl-dhheer
Out
Pixzstnct
252M
1.36
DecVQA
BookaCCrpps - C4 Wob HTMI.
DeeVig
google/gko2suue base!
gasglojjeccmmms-angg
2ooglelpocssmme docrga base'
posplopis2imutt-b doweiiit-ttt---e-aaaa-aeeee
Table 8: Details of the public checkpoints used as target models in this work. denotes checkpoint from Hugging Face.
Model
Optimizer
Learning Rate
Weight Decay
Batch Size
Scheduler
Iteration
VTS
Adam/
2o-4
200k
Donut
Adam
3e-5
0.01
linear warmup 10%.
800k
Pix2Struct-B B
AdaFactor
1e-5
warmup 1000 steps, cosine decay to O
800k
Table Details of the training hyperparameter each target model this work.
If public checkpoints are unavailable, we fine-tune the selected model the respective private dataset, using the pre-trained checkpoint as the initialization, with the training procedure outlined perform early stopping based on validation
by the respective authors. To prevent overfitting, performance, ensuring that all evaluated models generalize well to unseen data. We also use the pre-trained checkpoint initialize the proxy model F, train it Dauery- We provide overview
17

---

Published as a conference paper at ICLR 2025
Datasct
Model
Tost Set
ACC
ANLS
Train-Test Gap
Original
81.4
90.17
VTS
MIA
82.74
90.91
11.44
MIA-rephrased
77.59
85
PFL
Original
74.73
88.66
Domat
MIA
80.15
91 64
22.2
rep-rephrseed
70.46
80.96
Original
60.1
69.33
VTS
MIA
75.54
81 61.69
36.22
MIA rephrased
73.57
79.89
Original
Donut
MIA
78.55
83.42
39.78
DVQA
rephraadd
72.57
77.12
Original
Pix2Struct-B
MIA
64,42
79.95
25.8
MIA-rephrased
63.81
74.06
Original
64.53
74.12
Pix2Struct-L
MIA
73.91
82.71
22.11
MIA-rephrased
69.93
79.15
Table 10: Voc Performance of the target models on PFL and DocVQA dataset. Train- Test Gap is computed as the different of Doc VQA Accuracy between member/non-memberr documents. MIA denotes the attack evaluation set, which is subset randomly sampled from the original train/test set, MIA-rephrased is its variants with rephrased questions by LLM.
of the training procedure for each target model, based on the respective papers. These procedures were adapted fit our computational resources, outlined in Table 9
D.2 TARGET MODEL PERFORMANCE ON DocVQA
To ensure the utility of the target models for our experiments, we validated that the DocVQA per- formance of each model checkpoint closely matched the results reported in the respective papers. Table 10 presents the target models' performance across both Doc VQA datasets. We observe a clear train-test performance gap, particularly in smaller models, while the gap tends to narrow for more generalized models or with increased dataset size.
Model
Scooz: TA
Scoap-Ui UA
Scoup EAu
Loss-TA
GRADINSS UA
ScoesLoss- UA
Mie-Kel
Min-Kikea
ACC
ACC
ACC
61:00mm
00300s
07.81
02.81
50.87
00.07
00.07
Denur
23.35
75.34
68.45
16.77m
7L.38.
73.67
78.99
70.07.
08.66.1
70,85,
00.07
36.5
32.00
50.5
60.35
VTS
75.87
75:75
72:17k
25.176
28. 13
13.67
77.00
71.47a
07.64
75.5k
76822
71.0
75.16
66.67
74.56
Denaer
7347
79 50
77.47
78.57cm
68.50a
89.80k
31.81
53.08
77
73 9833
MUSIL
81.88u
47.0
Table 11: Results from Baseline Attacks.. Gray color indicate attacks conducted in the black-box setting. indicates methods requiring grey-box access. Results are reported based on five random seeds for KMEANS,, any. The methods with the best average performance across the two metrics are highlighted bold.
White-box
FL.
FLLORA
IG
ACC
ACC
F1
ACC
DVQA PFL
VTS
66.63um(+5.96)
(=117)
65.177.((+1.30)
70.52.a
ol.83.0+++1100
69.15..o (+-..51
Donut
77.96, ol ++-6633)
72.83pol+
77.94a (-661)
75.17ma(+4)
79.89e.(+8.06
75.67a.u(+0.5)
76.60h
77.31o m5..7).
74.8% -0.34)
76.88u. (+1.))
Donut
80.33 -0.177)
82.18oa(+100
80.0val
81. 31.93ool (-0.8) (-0883
80.33 (00.17)
82.3
o (.2)))
Pix2StructB
al.67oo(+2.54)
72.22
70.50n
71.90oa (+4.28)
(2.00s((+2.87)
(2.82a((5.155)
Table 12: White-Box: Main Results of DocMIA. Values in parentheses indicate the improvement (posi-
tive/negative) our proposed attacks compared to the SCORELOSS- -UAa- Compared to all baselines, the methods with the hest average performance across the two metrics are highlighted in bold. Results are re- ported over five random seeds.
18

---

Published as a conference paper at ICLR 2025
Mx24truct-
Pis2Struce
6. 05 1.0 3.0 5.5
0.
2.2
Dieunt
Figure 7: Membership Features against three different target mod- cls on DocVQA Dataset. Top: distribution of average loss over questions from target documents on cach target model. Bottom: SNE visualization of the features used in our proposed attacks.
Figure 8: Distribution of Aver- age Distance from member and non member documents.
DocVQA
PPL
DocVQA
PFL
Transted Gaz
Tran-ioo Store
Transieu Scare
Then-Rot Gap
(a) White-box
(b) Black-box
Figure 9: performance versus the Train-Test gap. The target models exhibit varying Train-Tee Test gaps, measured by the difference in DocVQA scores between member/non-memble documents. Our attacks remain effective even when the gap is small, with performance improving as the gap increases across most target models and datasets. In contrast, baseline methods show more variable performance under these conditions.
D.3 COMPUTATION AND RUNTIME
All attack methods are implemented using PyTorch and executed on NVIDIA GeForce A40 GPU with 45 GB of memory. The maximum runtime for each attack does not exceed 10 hours per run, depending on the target model's size and the pre-processing steps required for the data. This runtime reflects the efficiency of our approach, especially when compared to methods based shadow training, which require retraining of large-scale models many times to be effective (Carlini et al.) 2022). Our results demonstrate that the proposed attacks are both efficient and scalable, making them practical for large-scale models in real-world applications.
E MORE ON ATTACK RESULTS
In this section, provide the full results of all evaluated attacks in terms of Accuracy and Fl Score, summarized in Tables 11 and T2 In addition, we evaluate our attacks using TPR @ FFPR and TPR 3&FPR, following standard practices in recent MIA literature, as shown in Table B and
20
interesting observation the high performance of the LOSS-TA method for VT5 on DocVQA and Donut on PFL. in Table [3] This performance can be attributed to the clear separation in the loss distribution between member and non-member samples (Figure 7). which indicates overfitting behavior in these cases.
F MORE ON ANALYSIS
In this section, we provide a deeper analysis of the effectiveness of our proposed white-box and black-box attacks, highlighting their performance relative the baseline approaches.
19

---

Published as a conference paper at ICLR 2025
DVQA
PFL
VT5
Donut
Pix2Struct-B
VT5
Donut
1%
3%
1%
3%
1%
3%
1%
3%
1%
3%
LOss-TA GRADIENT- UA
7.67
14.00
0.67
7.67
2.33
5.33
0.67
3,00
1.67
14.67
2.33
9.33
3.67
6.00
1.00
5.00
0.33
3,00
1.00
8.33
SCORELOSS UAail
1.33
4.67
2.67
8.67
2.00
6.67
0.33
4.00
0.67
6.33
Min-K%
2.67
10.67
0.33
1.33
0.33
5.33
1.67
5.67
0.00
0.00
Min-K%++
1.00
7.00
4.33
9.33
0.67
10.33
1.00
8.00
0.33
2.00
FL
2.33
5.67
3.33
10.67
6.00
11.00
3.67
8.67
0.33
FLLORA
3.33
11.33
2.67
5.33
3.67
6.33
1.33
3.33
0.33
10.00
IG
0.67
5.67
1.33
8.00
3,00
10.33
1.00
2.33
5.67
11.00
Table 13: White-box: TPR at fixed FPR. Comparison across all white-box methods, with best- performance highlighted in bold. 1% and 3% indicate @ 1%FPR and TPR@ 3%FPR respectively.
F.1 IMPACT OF SELECTED FEATURES
As outlined in the main paper, we fix the set of selected features across all experiments. These features include the DocVQA score u, the optimization- based distance A, and the number of opti- mization steps s, aggregated using the set of aggregation functions aall {AVG; MIN; MAX; MED) We first evaluate the impact of individual features and their combinations on attack performance in the white-box DocMIA setting, using AVG the aggregation function tb The analysis employs the best hyperparameters identified during the tuning process described in Section
20151
VTS
Domut
AVG(NLS)
AV)
AVG(s)
AVG(NES) AVG(A) AVO(s)
AAEN
56.0
4630
0930
68.88
67.5%
2401
715
V
70.92
73.16
1125
4
71.09
5
72.87
Table Comparisons in Attack
71.11
73.67
71.22
73.86
Performance in terms of F1 Score:
(Top) between our Aptimization-haser Features with the loss value é (Eq. and the gradient norm ||Va (Bor- tom) between AVG and Qat as the ag- gregation functions.
Table 14: Impact of SSleeted Features PFL-DocVQA Models.
Donut
Pix2Stract-B
AVG(N AVG AVG
AVG(NES)
AV AVG(()
F1
AVG(NES)
AVG(A)
AVG(s)
F1
72.73
/
76.88
72.60
70.57
74.34
60.32
69.00
65.94
73.20
75.04
72.17
72.87
74.19
60.29
70.17
74.96
72.94
73.22
Table 16: Impact of Selected Features DocVQA Target Models. Only AVG is used the aggregation function Q. Attack performances are obtained with our FL method using the best hyperparameters.
Table 14 and Table summarize the attack performance when individual features their combina- tions are used. Additionally, Table (Top) compares the attack performance of our optimization- based features with the loss value & and the gradient norm of the loss with respect the model parameters 0. Here, the loss value é is computed uniformly across all target models over K genera- tion steps, given a (document, question, answer) example (ær, g, a) as:
log po [
(4)
20

---

Published as a conference paper at ICLR 2025
When used individually, our proposed optimization-basee features outperform the Doc score and the loss in most cases. Our attack methods are particularly effective against target models like VT5 and Donut trained PFL-DocVQA. which exhibit lower overfitting and small Train-Tes gaps (as shown in Tabl These results highlight that our attacks provide more discriminative features than the commonly used MIA features.
When combined, our selected features achieve the best or near-best performance across all cases. Furthermore, extending aggregation functions from AVG Pall adds notable improvements in attack effectiveness, as shown in Table [T| (Bottom). These results demonstrate that our proposed feature set is robust across different target models, making a reliable choice for DocMIA.
F.2 IMPACT OF THE TRAINING QUESTIONS KNOWLEDGE
So far, our document MI attacks against Doc models have assumed complete knowledge of the original training questions. We now relax this assumption and investigate how the lack of access to the exact training questions affects attack performance. In practice, an adversary may not have access to the exact training questions but can approximate them. For example, documents like invoices often follow standard layouts, and biases in human annotation may lead predictable patterns in the types of questions asked during the creation of DocVQA datasets (Tito et al.) [2024] Mathew et al.) 2021]. It is important note that the original training questions tend to simple, natural questions designed to extract specific information from the document. Moreover, the type of question is inherently linked to the type of document on which the DocVQA model trained. For instance, if the target model is trained on invoices, the natural type of question would focus on extracting essential details from the invoice, such as the "total amount", framed in a clear and straightforward manner e.g.. "What is the total?". This makes it possible for an adversary to generate approximate versions of the training questions, simulating a more realistic attack setting.
SCORE-TA
SCORE-UA,
LOss-TA
SCORELOSS-UAL
OURS (FL)
Model
ACC
FI
ACC
F1
ACC
FI
ACC
F1
ACC
Fl
VT5
60.67
64.13
55.830.0
46.8 89.8
54.50
59.19
55.83p.o
86.89..0
64.00ua
69.14kB
DVQA PRZ
Donut
69.17
69.72
59.33.g
51.59..o
68.50
66.67
59.17a.o
51.490.
71.13aas
72.07mM
VT5
73.67
75.01
74.830.o
74.366.o
71.67
74.06
75.17
74.96u.
74.830o
75.68mo
Donut
69.17
71.23
65.17
62.21o.0
52.33
33.57
65.17a.0
62.21g.
67. 67.67a.0
68.51a.0
Table 17: Results with Rephrased Questions. Gray color indicate attacks conducted in the black-box setting. All results are reported based on five random seeds. The methods with the best average performance across the two metrics are highlighted in bold.
DVQA
PFL
VT5
Donut
VT5
Donut
PFL
DVQA
1%
3%
1%
3%
1%
3%
1%
3%
Donut
Pix2Smmut-B
1%
3%
1%
3%
1%
3%
Min-K%
3.00
4.33
0.33
1.00
6.33
20.33
2.00
2.33
Min-K%++
3.00
4.67
0.00
2.67
6.33
10.00
0.00
7.00
SCORE-TA TA
0.33
2.67
3,33
9.67
3.00
8.67
SCORE-UAA
0.33
2.67
2.33
9.33
8.67
FL
0.67
5.00
3,33
8.00
3.67
17.33
3.00
4.67
FL
0.33
1.33
0.33
4.00
1.33
4.67
FLLORA
5,00
9.33
0.67
5.00
4.33
10.00
FLLORA
1.00
5.33
1.67
5.00
2.33
6.33
IG
5.33
8.00
1.00
5.00
5.33
8.00
1.67
10.00
IG
2.67
6.33
1.67
11.00
3.67
9.33
Table 18: Whitc-hox Results: TPR at 1% and 3% FPR Table 19: Black-box Results: TPR at 1% and with Rephrased Questions. Comparison to white-box meth- 3% FPR with Rephrased Questions. Donut is ods: Min-K% and Min-K%++ methods, with the best method used as The Proxy Model. in bold.
To explore this scenario, conduct experiments where we paraphrase the original training ques- tions using Mistral (Jiang et al. [202)), and use these rephrased questions as inputs for the MI attacks. As illustrated in Table 17] the performance of all MI attacks declines when rephrased questions used, mirroring the drop in Doc VQA model performance (Table 10), which is expected due to the increased uncertainty introduced by question rephrasing.
Among the baselines, the SCORE-TI TA attack proves particularly to be robust, especially against models trained on VOA, which show higher degree of overfitting. In contrast, attacks incorpo-
21

---

Published as a conference paper at ICLR 2025
DVQA
PEL
Targel
VTS
Donut
Pix2Struct-B
Pix2Struct-L
VIS
Donut
Proxy
1%
3%
1%
3%
1%
3%
1%
3%
1%
3%
1%
3%
SCORE- -TA
4.00
9.33
5.00
11.00
5,33
8.00
3.33
9.00
1.00
5.00
0.67
2.67
SCORE-UA
3.67
7.67
4.33
15.67
4,00
6.33
4.33
6.67
0.67
3.33
0.33
3.33
SCORE-UAMt
9.33
5.00
11.00
5.33
8.00
3.33
1.00
5.00
0.67
FL
0.67
12.33
11.67
23,00
2,00
16.67
2,00
5.33
0.67
2.00
5.00
8,00
VTS
FLLoRA
4.67
11.33
6.34
16.33
2.33
9.33
1.00
4.67
2.00
3.33
0.00
2.00
IG
8.33
2.00
7.00
4.67
7.67
2.33
0.33
3.67
1.33
6.67
FL
0.33
6.33
0.33
4.00
1,33
4.67
3.00
7.33
0.33
1.33
1.33
4.00
Donut
FLLoRA
1.00
6.33
1.67
5.00
2.33
6.33
3.00
8.00
0.00
5.33
2.00
5.33
IG
1.67
5.00
0.67
11.00
3.67
9.33
4.67
6.33
2.67
6.33
1.67
4.33
Table 20: Black-box: TPR fixed FPR. Comparison across all black-box methods, with the hest-performing methad highlighted in bold. 1% and 3% indicate TPR 1%FPR and TPR @3%FPR respectively.
rating loss-based signals introduce additional noise due to uncertainty, leading to a noticeable drop in performance.
Despite the rephrasing, our attacks remain effective, maintaining performance levels comparable to those observed with the original questions, especially against the two PFL models, which demon- strate a lower degree of overfitting.
We also evaluate our proposed attacks against other methods in this setting, focusing on TPR at 1% and 3% FPR, with the results summarized in Table 18 and 19
F.3 THE RESULTING PROXY MODEL
The purpose of training the Proxy Model on Deuery, with labels generated by the black-box model, is to mimic the prediction patterns of the black-box model. The expectation is that the proxy model can capture internal decision-making patterns by following the black-box's prediction strategies. Instead of optimizing for ground-truth labels, we train the proxy to maximize the likelihood of the generated labels. The training process concludes when the proxy achieves near-zero training loss, at which point the final checkpoint is used for the attack.
DONUT
Proky: VTS
Seral
Nireender
50
2.0
40
1.0
20
0.5
20
0.8
(a) Training curve
(b) Distribution over distance
Figure 10: The resulting Proxy Model against Pix2Struct-B B in the black-box setting. (a) The attack accuracy improves quickly once loss reaches near zero. (b) The optimization distance values between member and non-member documents exhibit separation similar to that seen the white-box setting.
As illustrated in Figure I0(a) the attack performance quickly improves as training progresses. The model overfits quickly, with attack performance reaching its peak early- after just a quarter of the training process demonstrating the efficiency of our approach. This suggests that once the proxy model converges, it has effectively captured informative membership signals from the black- box model, making it ready for the attack. Moreover, we compare the distribution of optimization distances between the proxy model and the same model in the white-box setting, as shown in Figure 10(b) The results show similar degree of separation between the two clusters in both cases, indicating the proxy model's effectiveness in approximating the black-box model's 's behavior to a certain extent.
22

---

Published as a conference paper at ICLR 2025
F.4 ATTACK PERFORMANCE AGAINST MINIMAL- TRAINING DOCUMENTS
Doc models rypically process each question-answer pair independently, resulting in multiple exposures of each document during training. This increases the likelihood of being memorized by the model, making such documents more vulnerable to MIAs. Intuitively, documents associated with fewer training questions should be less exposed and therefore be less vulnerable.
FL
IG
Model
nu 1(1)
m 2(1)
3(83)
ALL((30)
VT 1(1)
vn = 2(1)
3185)
ALL(300)
PHL
VT5
83.53
87.67
100
100
85.88
86.33
Donut
100
100
100
97.67
100
97.65
97
Model
1(51)
2(60)
3(52)
A1.1(300)
1(51)
2(60)
3(52)
AL.1(300)
DVQA
VTS
86.27
71.67
84.62
77.00
90.2
85
86.54
80.67
Donut
88. 24
73.33
76.92
77.33
56.86
68.33
55.77
61.33
Pix2Struct-B
90.2
93.33
90.38
87
88.24
88.33
76.92
73
Table 21: Membership Prediction Accuracy on Member Documents with minimal repetition. m denotes the subset of testing documents with m training questions, with subset sizes shown in parentheses. Compared the performance measured on the entire member set (denoted as ALL), attacks are still robust against documents with the low risk of memorization.
To evaluate this, we measure the accuracy of membership predictions from our attacks on a subset of member documents in Diest associated with only few training questions. These documents represent a minimal memorization risk, posing more challenging evaluation scenario. Results in Table 21] show that our attacks remain effective on these subsets, achieving high accuracy even for documents m = 1 training question. This demonstrates the robustness of our attacks under conditions of minimal repetition.
G DEFENSES
To mitigate the privacy vulnerabilities associated with membership inference attacks in Document Visual Question Answering (DocVQA) systems, we can employ Differential Privacy (DP) tech- niques (Dwork al. 2014). specifically through the use of differentially private stochastic gradient descent (DP-SGD) introduce by [Abadi et al.) (2016). DP is a robust framework that ensures an in- dividual's data contribution cannot be inferred, even when an adversary has access to the model's outputs. DP-SGD achieves this by adding calibrated noise to the model's gradients during train- ing, thus providing strong theoretical privacy guarantees. However, this approach is not without its drawbacks; the necessity of noise injection can adversely affect the utility of the trained model, leading to reduced performance in answering queries accurately. Alternatively, we can consider ad-hoc solutions such limiting the number of queries setting, which would inherently reduce the model's 's usability and flexibility in practical applications. While these measures can enhance privacy, they also necessitate careful consideration of the balance between privacy protection and the functionality of DocVQA systems.
question per document in black-box
ANLS
Fl
TPR
ANLS
TPR 33FPR
ANLS
F1
TPR Ø 3S&FP
FL
55.09
2.33
58.84
4,33
73.81
7.33
FLLORA
19.16
54.94
2.00
21.81
58.94
3.67
50.12
73.81
7.33
IG
56.29
1.67
59.35
5.00
73.52
8.67
Table 22: DocMIA Results for Donut trained with DP-SGD on DocVQA dataset. We report the attack performance of our FI. method in terms of Fl score and FPR.
To evaluate the robustness of our proposed membership inference attacks against Differential Pri- vacy (DP), we implemented the well-known DP-SGD algorithm. We considered five privacy budget E E {8, 32), with corresponding noise multiplier a E (0.5767822266, 0.3824234409), respectively. The composition of the privacy budget over multiple iterations caleulated using Rényi Differ- ential Privacy (RDP). We then converted the RDP guarantees into the standard (c, 6D notion following the conversion theorem from (Balle al. 2020
23

---

Published as a conference paper at ICLR 2025
We trained the Donut model on the DocVQA dataset with DP-SGD provide theoretical privacy guarantees for individual training documents. Due to resource constraints, we resized document resolution to smaller size (1280, 960) compared to (2560, 1920) in the public checkpoint provided by the original authors, which slightly reduced the model's Doc performance. For additional refer readers to the original model's paper(Kim
details the effects of document resolution, iet al. 2022). The model was trained using the Adam optimizer with learning rate of le 4, for 10 epochs, and with a batch size of 16. DocVQA performance was evaluated using the Average Normalized Levenshtein Similarity (ANLS) metric.
Table 22 summarizes the results. As expected, introducing DP into model training significantly reduces the attack performance, for example from 73.81% Fl score with non-DP model to 55.09% & 8, but this comes at the cost of substantial utility degradation, with the DP model achieving less than half of the performance of the non-DP model, 21.81 of ANLS at = 8 compared to 50.12 of ANLS from non-DP checkpoint. For higher privacy budgets (e 32), our attacks demonstrate improved effectiveness, achieving notable gains, +3.75 in Fl and +2 in TPR3%FPE scores compared to 8, as the model becomes less privacy-constrained.
24


Reviews:
[{"review_id": "moeOfQppiE", "reviewer": "Reviewer_HkZx", "summary": "This paper presents membership inference attacks tailored to Document Visual Question Answering (DocVQA) models, highlighting potential privacy vulnerabilities in handling sensitive document data. It introduces white-box and black-box attack methods that work without relying on auxiliary datasets, offering practical insights into real-world privacy risks. The attacks utilize features that typically arise from training data exposure. By addressing the unique complexities of multimodal data, these attacks outperform current baselines, underscoring the need for improved privacy safeguards in DocVQA systems.", "questions": "- Given the assumption of repeated document exposure, how would the attacks perform in settings where training data consists of unique or one-time documents, with minimal repetition?\n- What are some potential countermeasures or defense strategies to mitigate the privacy risks highlighted by these membership inference attacks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Fills a critical gap in privacy research for multimodal AI applications by designing MIA for DocVQA\n- By designing attacks that operate without auxiliary datasets, the paper presents a more realistic and practical approach for assessing privacy risks in scenarios with limited data access\n- The attacks can be employed in both whitebox and blackbox attacks settings\n- The attacks use intuitive optimization-based discriminative features", "weaknesses": "- The attacks rely on repeated instances of documents in the training data, which may not always be present in real-world DocVQA applications, potentially limiting the generalizability of the approach\n- While the paper identifies privacy risks, it lacks a discussion or evaluation of potential defenses that could mitigate these vulnerabilities", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WlzJdbMiTR", "reviewer": "Reviewer_i62x", "summary": "This paper studies the document-level membership inference attacks targeting multi-modal models for document visual question answering. The main idea is to extract discriminate features between member and non-member documents based on their optimization trajectory. For black-box settings, a proxy model is utilized to extract discriminate features that are relevant for the black-box target model. Empirical results show that the proposed approach consistently performs well in various settings.", "questions": "Having additional ablation studies as well as providing more insights on the asymmetric transferability between different proxy and target models will improve the paper.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of leveraging optimization trajectory for membership inference is novel. \n2. The empirical performance of the proposed method is stable, indicating a good generalization over different attack settings.", "weaknesses": "1. There is a lack of ablation study on the choice of the features. In particular, giving some insights about what features are most useful for learning discriminative features between member and non-member documents. Of course, a conclusion of all obtained features being equally important or specific features being more relevant for certain applications are also good observations. \n2. There is no clear explanation on the asymmetric transferability between different proxy and target model pairs Table 2. I think these are all key observations that demystify the success of membership inference attacks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BC74OcZnXe", "reviewer": "Reviewer_hZiB", "summary": "This paper introduces Document-level Membership Inference Attacks (DocMIA), focusing on Document Visual Question Answering (DocVQA) models. The paper presents two novel membership inference attacks tailored for DocVQA, targeting both white-box and black-box attack scenarios. These attacks enable adversaries to determine whether a particular document was part of the training set, even in the absence of auxiliary datasets. The proposed attacks leverage unsupervised optimization-based methods that outperform existing state-of-the-art membership inference attacks, underscoring the privacy vulnerabilities present in DocVQA models.", "questions": "How does DocMIA perform when there is a significant distribution shift between the public and private datasets? For instance, if the private dataset is derived from a financial dataset, but the public dataset stems from a different domain (e.g., legal documents), how would that affect the attack’s efficacy?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The focus on DocVQA models for membership inference is unique. Previous research has mainly centered on generic models or other multimodal domains, while this paper contributes significantly to understanding privacy risks specific to document processing and VQA models.\n- The paper employs solid, theoretically grounded approaches to devise the DocMIA attacks. The white-box attack uses optimization-based discriminative features, allowing more robust membership inference without requiring shadow models. The experimental setup includes multiple DocVQA models (VT5, Donut, Pix2Struct), ensuring that results are thorough and comparable.\n- Given the sensitive nature of document data, this paper’s contributions are valuable for highlighting vulnerabilities in DocVQA models. The presented attacks have significant implications for the adoption of such models in privacy-sensitive applications, urging the development of more robust defenses.", "weaknesses": "-  While the paper references relevant literature, it does not include direct empirical comparisons with existing black-box defenses. Including black-box defenses as baselines would allow a more comprehensive assessment of DocMIA’s performance, helping to contextualize its strengths better.\n- The experimental setup lacks some critical details, such as the specific training dataset used for GAN priors in particular baselines. Information on hyperparameters for certain attack baselines (e.g., gradient-based attacks) would improve reproducibility.\n- The paper does not address the performance of DocMIA under dataset distribution shifts. This gap is significant as DocVQA models may be deployed across varied domains, potentially affecting the reliability of the proposed attacks. Testing DocMIA under different dataset shifts would provide insights into its robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OkEY8L3QTZ", "reviewer": "Reviewer_3nRY", "summary": "The paper aims to make a fine-grained membership inference attach, which can determine whether a sensitive document is included in the training dataset. For the white box and black box, the authors have developed different MIA approaches according to their characteristics. Experimental results confirm that their methods outperform baselines.", "questions": "1. Briefly describe how to solve “shadow training of proxy models becomes infeasible\".\n2. While the paper introduces methods to adapt the attack to black-box models through knowledge transfer, this approach may not be as effective as direct white-box attacks due to the inherent limitations in approximating the behavior of complex models.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper is the first attempt to target document-level membership inference attacks.\n2. This paper designs attack modes for white box and black box respectively.", "weaknesses": "1. The challenges of the work are underdescribed by the authors, and it is difficult for the reviewer to understand the technical challenges between fine-grained inference attacks and coarse-grained attacks, proposed by Tito et al.. The authors express it more as if they were writing an experimental report.\n2. \"shadow training of proxy models becomes infeasible.\", the reviewer suggests the authors describe in words how to solve this problem and highlight it.\n3. While the paper introduces methods to adapt the attack to black-box models through knowledge transfer, this approach may not be as effective as direct white-box attacks due to the inherent limitations in approximating the behavior of complex models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QX1IosM4pO", "reviewer": "Reviewer_7wUB", "summary": "This paper introduce the first document-level membership inference attacks Document-Level Membership Inference Attacks (DocMIA) for Document Visual Question Answering (DocVQA) models, addressing privacy risks in multimodel contexts. Through extensive experiments across multiple datasets and models, significantly outperform existing membership inference baseline.", "questions": "1、The paper presents experiments on four target models for the DocVQA dataset but only conducts experiments on two models for the PEL-DocVQA dataset, without providing an explanation for this choice of setup. This lack of clarification on the experimental design for the PEL-DocVQA dataset leaves the rationale for the selection of models unclear. It is recommended to include an explanation for this experimental setting.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1、This paper proposes Document-Level Membership Inference Attacks (DocMIA), which deal with the multiple appearances of the same document in the training set.\n\n2、DocMIA consistently achieves high performance across all target models, the strong performance of  DocMIA highlights the privacy risks posed by optimization-based features of membership.", "weaknesses": "1、The writing in this paper is clear and accessible, providing a well-structured and understandable presentation of the authors' ideas and methods. However, the text is overly verbose, with an excess of written content and a lack of visual aids. The absence of a diagram illustrating the attack process detracts from an intuitive understanding of the attack methodology. Adding visual elements would enhance the reader's comprehension and provide a more direct insight into the core processes discussed.\n\n2、There are some typographical problems with the manuscript. The position of Figure 1 is not ideal and should be adjusted to be centered side-by-side, similar to Figure 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hplmopEkFz", "reviewer": "Reviewer_V2Bv", "summary": "This work tackles document-level inference for VQA models that have been trained on text as well as visual-documents, where additional processing steps like OCR and repeated instances of documents make the problem significantly different from standard MI under language models. The authors propose attacks that are free of any auxiliary data, for both black-box and white-box access models.  The proposed attacks are evaluated on multiple datasets, outperforming both existing and newly-designed baseline inference methods.", "questions": "- L106: why is the adversary query-restricted?\n\n- Section 4.2.1: How is convergence defined here? There is some (unintentional) leakage happening, since you need to know when \"convergence\" happens for members to stop the optimization at that point. A true test to check for this would be to run it for a pre-defined set of iterations, record the loss at each iteration, and use that as the feature vector.\n\n- L354: How are gradients back-propagated via the OCR generation pipeline?\n\n- Section 5.3: How are the questions constructed? Are they picked based on what was in train/test data?\n\n- L410: What is \"predicted labels\"? Isn't the task to generate an answer? If so, just say 'generated text'\n\n- L1008: \"...paraphrase the original training questions\" - do their answers remain the same?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The work tackles document VQA systems, which post a lot more challenges than simple text-based language models. Understanding leakage in this scenario would be helpful.\n- The attack methodology is interesting and straightforward, and the authors experiment with multiple ways of \"fine-tuning\" the target model to measure scores used for membership inference.\n- Results are presented very well, and overall the figures/tables are very neat, with a very detailed Appendix.", "weaknesses": "I think the works make a fair contribution, but I do have some concerns (major and minor listed below and under questions) that the authors could answer to help me better understand some of their design choices.\n\n- The evaluation design is completely disconnected from how MI attacks are usually evaluated. Currently, thresholds are computed based on average values in $D_\\text{test}$. In MI (or similar evaluations), the standard protocol is to sweep over the non-member's values and based on the FPR (false positive rate) dictated by a certain threshold, compute the TPR (true positive rate). From there, multiple statistics (such as AUC for the TPR-FPR curve, or TPR at a specific FPR) can be computed. I would request the authors to re-evaluate their methods using this approach. I also do not know why the authors have constructed their own version of baselines- there are existing attacks (as one example, see Min-k%++)\n\n- Appendix E.1: \"... have assumed complete knowledge of the original training questions\" This is a very strong assumption! This implies that the adversary also knows the exact questions associated with the target document (and by extension the answers). This was not mentioned clearly anywhere in the main paper. The fact that performance drops when this is relaxed is not surprising to me, and demonstrates that the majority of performance reported in the paper is originating from this strong (and obfuscated) assumption. The relaxed setting here (where exact questions are not known) should be the **default** setting in the main paper- after all, true \"document\" inference would not entail one knowing which *exact* questions and answers were present in the training data\n\n\n## Minor comments\n\n- L31: \"...fuels a significant number of operations daily\" - source?\n- L49-50: \"...utilize a dual representation...\" - please cite relevant works\n- L102-102: \"The literature indicates that....\" - this is not the case. While recent work [1] does suggest that parameter access may be necessary for optimal membership inference, the cited papers here (and most empirical results) indeed conclude that additional access to parameters does not help with membership inference. \n- L114: \"...larger scale models\"- please see [2] that indeed tackles inferring the use of certain text documents for LLMs.\n- L186: \"adversary lacks access to an auxiliary data\". Membership-inference evaluation by design needs \"non-members\" to compute thresholds for membership classification (and as recent work [3] has shown, you cannot just use any non-member data especially when it comes to language). What the authors probably mean here is that there is not *enough* auxiliary data, and should clarify this.\n- L191: \"...would be prohibitively expensive\" - this would be a problem for a resource-constrained adversary, but an adversary that is malicious (in the security sense) will not be limited like this. An adversary can always use public/pretrained models as a starting point.\n- L210: \"...types of documents\" - define knowledge of \"type\" of documents/questions\n- L447: \"Table 1 (right)\" - there is only one table. Please fix reference (similarly for Table 1 (left), etc.)\n\n### References\n- [1] Suri, Anshuman, et al. \"Do Parameters Reveal More than Loss for Membership Inference?\" HiLD, ICML (2024).\n- [2] Maini, Pratyush, et al. \"LLM Dataset Inference: Did you train on my dataset?\" arXiv:2406.06443 (2024).\n- [3] Duan, Michael, et al. \"Do membership inference attacks work on large language models?\" arXiv:2402.07841 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)