Gradient Inversion of Multimodal Models
Authors: Omri Ben Hemo
Alon Zolfi
Oryan Yehezkel
Omer Hofman
Roman Vainshtein
Hisashi Kojima
Yuval Elovici
Asaf Shabtai
Venue: ICML 2025 poster (2025)
Keywords: Privacy
Gradient Inversion
Federated Learning
Multimodal
Vision-Language
Document Question Answering
Primary area: social_aspects->privacy

Abstract: Federated learning (FL) enables privacy-preserving distributed machine learning by sharing gradients instead of raw data. However, FL remains vulnerable to gradient inversion attacks, in which shared gradients can reveal sensitive training data. Prior research has mainly concentrated on unimodal tasks, particularly image classification, examining the reconstruction of single-modality data, and analyzing privacy vulnerabilities in these relatively simple scenarios. As multimodal models are increasingly used to address complex vision-language tasks, it becomes essential to assess the privacy risks inherent in these architectures. In this paper, we explore gradient inversion attacks targeting multimodal vision-language Document Visual Question Answering (DQA) models and propose GI-DQA, a novel method that reconstructs private document content from gradients. Through extensive evaluation on state-of-the-art DQA models, our approach exposes critical privacy vulnerabilities and highlights the urgent need for robust defenses to secure multimodal FL systems.

Gradient Inversion of Multimodal Models
Omri Ben Hemo
Alon Zolfi
Oryan Yehezkel'
Omer Hofman? 2
Roman Vainshtein
Hisashi Kojima*
Yuval Elovici'
Asaf Shabtai
Abstract
Bhinng Cydl
Bilking Calendar
Frade
Broadcast
Prspary
Account Executive
Federated learning (FL) enables privacy- preserving distributed machine learning by sharing gradients instead of raw data. However, FL remains vulnerable to gradient inversion attacks, in which shared gradients can reveal sensitive training data. Prior research has mainly concentrated on unimodal tasks, particularly image classification, examining the reconstruc- tion of single-modality data, and analyzing privacy vulnerabilities in these relatively simple scenarios. As multimodal models are increasingly used to address complex vision-language tasks, it becomes essential to assess the privacy risks inherent in these architectures. In this paper, we explore gradient inversion attacks targeting multimodal vision-language Document Visual Question Answering (DQA) models and propose GI-DQA, a novel method that reconstructs private document content from gradients. Through extensive evaluation on state-of-the-art DQA models, our approach exposes critical privacy vulnerabilities and highlights the urgent need for robust defenses to secure multimodal FL systems. Project page at: https:/ hatpa:://AAozoofff wtt.tt.......iopp///eoo
WPVI
Arne Ciiford
Philadlphii
Peppert
Execulive
Sale Offic
Kalkigg Cysle
Property
WPMM
Anne Cliffedd
PrMlodd Tothes NIT
Figure Gradient inversion attack on DQA models in FL. Left: document with private data from benign client. Right: zoomed-in section shown in three versions (top to bottom): original, adversary's 's reference (public document template), and GI-DQA reconstruction reconsrucionnddemossrriiiig demonstrating private data recovery using only gradients and a document template.
these gradients to update the global model, reducing pri- vacy risks by avoiding the exchange of raw data (Banabilah et al., 2022). This paradigm has been successfully applied in real-world scenarios, including Google Keyboard (Yang et al., 2018), Apple's Siri (Paulik et al., 2021), and criti- cal domains such healthcare (Thrasher al., 2023) and finance (Shi al., 2023).
Despite its privacy-preserve design, FL, remains vulner- able to gradient inversion attacks (Du et al., 2024), which exploit the gradients shared by clients to reconstruct sensi- tive information from clients' private training data. Prior research has primarily focused on recovering private im- age data using optimization-base methods, including iter- ative optimization to minimize gradient differences (Zhu et al., 2019; Zhao et al., 2020), enhancements through co- sine similarity and regularization (Geiping et al., 2020), and leveraging batch normalization for improved recovery in batches (Yin et al., 2021). More recent efforts have extended these attacks to complex architectures such as vision trans- formers (Lu al., 2021: Hatamizadch al., 2022). While gradient inversion attacks have been extensively studied in unimodal tasks like image classification, their potential impact on multimodal applications remains unexplored.
1. Introduction
Federated learning (McMahan et al., 2017) has emerged as a popular paradigm for privacy-presed preserriee distributed ma- chine learning. In FL, multiple clients collaboratively train a global model under the coordination of central server over several rounds. During each round, clients update the local model using their private training data and transmit only the computed gradients to the server. The server aggregates
Equual contribution 'Ben Gurion University of the Negev, Israel FFujitsu Research of Europe Fujitsu Limited. Correspondence to: Omri Ben Hemo/ / Alon Zolfi < <omribenh.zooff) @ post. bgu.aciu ac..a...l>.. il?.
Recently, multimodal models have gained significant atten- tion for their ability to process and integrate diverse sources of information, including text, images, and spatial features,
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Gradient Inversion of Multimodal Models
enabling improved performance across a wide range of vision and language tasks (Wang et al., 2023). Among these, DQA models stand out for their focus on addressing document-related queries by leveraging multimodal capa- bilities. Broadly, DQA models can be categorized as either Optical Character Recognition (OCR) approaches. OCR- models, such as the LayoutLM family (Xu et al., 2020a;b; Huang et al., 2022), rely on the extraction of textual information prior to processing and the combination of it with spatial and visual representa- tions to achieve sfate-of-the-art performance. In contrast, OCR-free models, such as Donut (Kim et al., 2022), bypass external OCR modules by directly learning from document images. Compared to unimodal models, which process only single type of data, multimodal models excel at captur- ing intricate cross-modality relationships, offering richer semantic and contextual understanding. This capability not only enhances their effectiveness in handling sophisticated applications, but also underscores their growing prominence in cutting-edge research. Despite their effectiveness, train- ing multimodal models requires access to large, privacy- sensitive datasets that often contain confidential text and images. Centralizing such data for training poses privacy risks, regulatory concerns (e.g., GDPR, HIPAA), and logis- tical challenges. FL offers a promising solution by enabling multimodal models to be trained collaboratively across de- centralized data sources without exposing raw data (Nguyen & Karatzas, 2024).
words in document images is significantly harder than re- covering large objects in natural images. Unlike large-scale structures that provide strong spatial cues, fine-grained text is densely packed, making even slight variations in gradients highly disruptive to reconstruction, (ii) text-based gradients are inherently sparse, which limits the amount of recover- able information. This sparsity makes it difficult to distin- guish meaningful features from noise, further complicating precise recovery. These factors make DQA-specific gradient inversion uniquely challenging and require more advanced techniques than those used in standard vision-based attacks.
or OCR-free
To evaluate the effectiveness of our proposed method, we conduct extensive experiments on state-of-the-arr DQA models, including both OCR-based and OCR-free archi- tectures. Our evaluation demonstrates the feasibility of reconstructing private data with high fidelity, shedding light on the weaknesses of existing privacy-preservee nisms in multimodal FL setups. For example, on the Donut model, our attack is able to perfectly reconstruct 70. 1% of the documents' words.
mecha-
Our contributions can be summarized as follows:
To the best of our knowledge, we present the first gra- dient inversion attack on multimodal models, introduc- ing a novel method specifically tailored for multimodal DQA models.
A thorough evaluation of privacy vulnerabilities in state-of-the-ar OCR- OCR-based and OCR-free DQA archi- tectures.
Despite the increasing popularity of multimodal models, gradient inversion attacks on them remain significantly derexplored, particularly in decentralized learning environ- ments such as FL. To address this gap, we present compre- hensive study of gradient inversion attacks targeting DQA models. DQA is particularly interesting due to the sensitive and private nature of the documents it processes, such as medical records, financial statements, and legal contracts, making privacy concerns especially critical in real-world applications. Specifically, we propose GI-DQA, a novel method that reconstructs sensitive document content (e.g., personal identifiable information) from shared gradients in an FL setup. To reflect realistic attacker capabilities, we as- sume access to standardized document templates, which are common in real-world domains (e.g., invoices, and medical forms). These templates follow fixed layouts with only a few variable fields, allowing the attacker to focus reconstruc- tion efforts on regions likely to contain private information, as shown in Figure 1. By leveraging the unique structure of multimodal models, GI-DQA exposes critical privacy vulnerabilities inherent in these models, providing deeper in- sight into the risks associated with multimodal architectures and emphasizing the need for robust countermeasures pro- tect sensitive information. DQA-specific gradient inversion presents unique challenges: (i) reconstructing small-sized
Insights into the unique challenges of performing gradi- ent inversion on multimodal tasks, highlighting critical areas for future research.
An exploration of potential defense mechanisms to mitigate privacy risks, offering a roadmap to enhance the robustness of multimodal models in FL systems.
2. Background & Related Work
2.1. Gradient Inversion in Federated Learning
Gradient inversion attacks in FL. aim to reconstruct private data by exploiting shared gradients. Prior attacks can be categorized by: (i) server's trustworthiness:: adversaries lo- cated on the server may be honest-but-curious (Zhu et al., 2019; Zhao et al., 2020; Geiping et al., 2020; Hatamizadch et al., 2022; Lu et al., 2021), passively analyzing gradients without interfering, or malicious (Fowl et al., 2021; Chu et al., 2022; Pasquini et al., 2022; Fowl et al., 2022; Wen et al., 2022), actively disrupting the training process; and (ii) attack strategy: in optimization-batt based (Geiping et al., 2020; Yin et al., 2021; Hatamizadch et al., 2022; Gupta et al., 2022; Lu et al., 2021) attacks, the tround-truth sam- ples are approximated through iterative refinement, while in analytic- (Fan et al., 2020; Zhu & Blaschko, 2020)
2

---

Gradient Inversion of Multimodal Models
attacks, systems of equations are solved between gradients and inputs precisely reconstruct the data. In this work, we address the more challenging and realistic honest-but- curious setting, leveraging a combination of optimization- and analytic-based strategies.
et al., 2022), incorporate reading-oriented pretraining objec- tives to directly process document images, offering competi- tive performance while reducing dependency OCR tools. Both -CR-based and OCR-fre approaches highlight the diverse methodologies in DQA, addressing trade-offs be- tween interpretability. robustness, and performance across various document understanding tasks.
Most prior research has focused on recovering private im- age data through optimization- approaches. Zhu al. (2019) (DLG) first formulated the attack as an iterative op- timization problem, where attackers restore data samples by minimizing the distance between shared gradients and dummy gradients. Zhao et al. (2020) (iDLG) proposed extracting the label of a single sample from the gradients, which further improved the attack. Geiping et al. (2020) (In- verting Gradients) reconstructed higher-resolution images from ResNet-based models (He et al., 2016) by modifying the distance metric to cosine similarity and incorporating total variation as regularization term. Yin et al. (2021) primarily focused on recovery for larger batch sizes. Their method assumes access to normalization (BN) statistics, which enabled partial image recovery at larger batch sizes. Hatamizadeh et al. (2022) (GradViT) and Lu al. (2021) (APRIL) extended these attacks vision transformers.
3. Method
3.1. Preliminaries
DQA Federated Learning. A FL framework for DQA consists of a server that coordinates K clients, each with private dataset D₁ of N samples (x, v), where I represents the input data (visual document uD and question zq),, and y the target (corresponding answer). Clients collaboratively train a global DQA model fo with weights 0 without ex- changing raw data. In each round t, the server sends the current global model fe, to the clients for local training, for total of T rounds. Each participating client computes the average gradient VxO, on its local data using the current global model:
These works primarily address unimodal classification mod- els, where reconstructed images typically contain single prominent object. In contrast, our study focuses on an under- explored domain: the reconstruction of visual data from mul- timodal DQA models, where images consist of numerous fine -ma-ird textual elements, introducing unique challenges for inversion. A recent related work (Liu et al., 2024) consid- ers gradient inversion in a multimodal FL. setting: however, it employs independently trained, modality-specifft models without shared representations, coordinated through mutual knowledge distillation. This approach fundamentally dif- fers from our setting, which involves deeply fused, jointly trained architectures typical of modern multimodal models.
V = DD (V 2, v)
(1)
where C(-) represents the standard DQA training loss. Fi- nally, the server aggregates the collected gradients and up- dates the global model as follows:
K
5o141 + fo, + 17
VnOe
(2)
where v) is the learning rate.
Gradient Inversion. Given a gradient VaO, computed by
client & using the original data (ar, E DR. an adversary generates randomly initialized data (f, 9) which are itera- tively updated to approximate (ar y) by solving the follow- ing optimization problem:
2.2. Document Visual Question Answering
DQA has emerged as a prominent area of interest in the machine learning research community due to its ability to answer natural language questions by extracting meaningful information from documents. The importance of efficient management of document workflows spans multiple indus- tries, including banking, insurance, public administration, impacting virtually all aspects of business operations. A key differentiating factor among DQA models is their re- liance OCR technology. OCR-based models, such as the I LayoutLM M family (Xu et al., 2020a;b; Huang et al., 2022), TILT (Powalski et al., 2021), and UDOP (Tang et al., 2023), use OCR to integrate textual, layout, and visual features, achieving impressive performance. In contrast, OCR-free models adopt end-to-end architectures that bypass external OCR modules. These models, including Donut (Kim et al., 2022), Pix2Struct (Lee al., 2023), and Dessurt (Davis
:- - min di (VL(0, 2. 1). v.0,) + Ea,R,
(3) agR,
where d is distance metric (e.g., euclidean distance), R represents regularization term (e.g., total variation (Cham- bolle al., 2010)), and is weighting factor. Note that an adversary can perform the attack in any iteration E T.
3.2. Threat Model
Adversary's Capabilities. We consider an honest-but- curious adversary (Section 2.1) who has access to the gra- dients returned by an arbitrary client k, either by being physically located on the server or by eavesdropping on the communication between the central server and the client. At iteration i, the adversary has access to: (i) the gradients
3

---

Gradient Inversion of Multimodal Models
Benign Cvent
Document.
Ground-TrubhAnaawe
"Phoortiea Asttonn
DOA Modal fs
Predicted Answee
Gracients PO.see
"Prior USA"
DOA
Server
Quertion
Whet the advartion'' namme""
Moddal Beckpmasgetiion
Advarsary
DooumontTenppake &n
Reconstructid Artswar "Frorrtiee LEAA Action"
DQA Model
Predicted Anower
Adversery Gradients Viang
Advus Adverasry Loia
Eqrae Plarer)
LOWE
Reconstructed Question "Whot the adveriser'ss name?"
Altisceet Backk wwwpppopppoooooooo and piedis updatt
Figure 2: Overview of the FL setup with a benign client and honest-but-curious attacker. The benign client computes gradients their private document and shares them with the server, where the attacker is positioned. The attacker, equipped with document template (without private data), computes their own gradients and optimizes a loss function comprising gradient matching and auxiliary priors to reconstruct the client's 's document.
V&O, shared by the client, (ii) the current model parame- ters 0,, and (iii) a collection of publicly available document image templates (e.g., healthcare forms), which represent standard formats with private fields (such as names, dates, or amounts) left empty. These templates provide additional structural and contextual information that closely resemble the input documents, significantly narrowing the adversary's search space. This allows the adversary to focus on specific regions of interest within the documents, such as fields where private information is typically located. In many document-basee applications, such as invoices, medical forms, boarding passes, or receipts, the layout and struc- ture are highly standardized across users, with only small number of sensitive fields varying. Therefore, this setup is practically grounded and reflects realistic scenarios. Fig- ure 2 presents an overview of the attack pipeline.
Prior work on image classification models has demonstrated that using the ground truth label y (rather than randomly initialized one) significantly improves the quality of recon- struction (Zhao et al., 2020). This improvement is achieved by leveraging the negative sign traces of the gradient in the classification head. The enhancement arises because the computed gradients depend on the loss function, which is di- rectly influenced by the ground-truth label, thereby reducing the distance between the adversary's 's and client's gradients and simplifying the optimization process. However, in our case, as we employ question-answering models, this ap- proach is not applicable. Furthermore, the input x in our setting comprises both the document zp and its correspond- ing question zq- The input question TQQ also affects the calculated gradients, as it provides contextual information that guides the model to understand the relationship between the document FD and the target answer y. Consequently, reconstructing both the question zq and the answer y is crit- ical for the success of the attack (discussed in Section 4.2.2). To achieve this, build on previous works that propose analytic-based methods to first extract the question-answer pairs (Petrov et al., 2024; Deng et al., 2021; Balunovic et al., 2022). Once these pairs are reconstructed, we then optimize the document reconstruction to refine the overall results.
Adversary's Objective. The adversary aims to reconstruct the private fields contained within the client's documents. These private fields may include personally identifiable in- formation (PIT), such names (e.g., patients), ID numbers (e.g., SSNs), financial details (e.g., account balances), and sensitive textual data (e.g., medical diagnoses). The adver- sary seeks to exploit both the structure of the document (i.e., templates) and contextual information encoded in the shared gradients. This combination of prior knowledge and gradi- ent leakage significantly increases the likelihood of privacy breach, posing serious threat client confidentiality.
3.3. 1. DOCUMENT RECONSTRUCTION
For visual text reconstruction, we propose the following optimization objective:
3.3. Multimodal Gradient Inversion Attack
2 = arg min Lyrad VC Vx6.) + Rax)
In this section, we describe our proposed approach, which consists of two components: document reconstruction and text reconstruction. The document reconstruction follows optimization-bant based approach, while the text reconstruction employs an analytic-based approach.
Ranx Orx Rext + agaa R Rgau RTVRRV
(4)
where Lgrad represents the matching loss, and Raux is a set of auxiliary priors. The scheduling term (o(t) dynamically

---

Gradient Inversion of Multimodal Models
adjusts the influence of the auxiliary priors throughout the optimization process. We incorporate three auxiliary priors to improve reconstruction fidelity: (i) a visual text prior based on the Laplacian filter, which enhances fine-graing fine-grained text boundaries, (ii) a Gaussian prior, which encourages spatial smoothness and reduces high-frequency noise, and (ii) a total variation (TV) prior, which combines spatial smoothness with a channel-wise consistency constraint that encourages grayscale reconstructions, aligning with the typ- ical appearance of text and background in documents. In the remainder of this section, we describe each prior and scheduler in detail.
convolution operation with a fixed kernel:
Rxx(aD)
I (Kap tap * xiD)(
a
j
0
-1
0
Knp
-1
4
-1
0
-1 0
where Kaap is the 4-neighbor Laplacian kernel, (Kup (i,) is the convolution operation at a specific pixel (i, and [z'] is the total number of pixels in the image.
Gaussian Prior. To enhance the reconstruction quality of fine-grained details, we introduce a Gaussian prior, which leverages the properties of Gaussian smoothing filter to regularize the optimization process. The Gaussian prior enforces spatial consistency by penalizing high-free feeeffreuency noise while preserving key structures, such as edges and text regions, acting as a low-pass filter. Similar to the Laplacian filter, the Gaussian filter is implemented using a convolution operation with a Gaussian kernel:
Gradient Matching. The gradient matching loss serves as the primary component of the optimization, minimizing the discrepancy between the client's gradients and those of the adversary. Previous works have used distance metrics such as Euclidean distance and cosine similarity. In our approach, we combine both metrics for improved effectiveness: Eu- clidean distance is used to account for the magnitude of the gradient vectors, while cosine similarity attends to their directional alignment. This loss is defined as:
Rgna(27) (TT) =
ΣΣ
(K gau
- - (t,,,)
Lgrad CuSE CUSE ('L( 2), ().
82 4
Kgau (i.
exp
2TT
20
Lcs V'L 2, v.O.)
(5)
(8)
By applying the Gaussian filter to the reconstructed image and computing the difference with the unfiltered one, this prior quantifies deviations in smoothness. This encourages the optimization to focus on preserving the natural smooth- ness of the input image while avoiding overfitting noise artifacts. The Gaussian prior is particularly effective for ensuring smoother transitions in reconstructed document regions, facilitating a more realistic and accurate recon- struction. Compared to the commonly used total variation prior (Chambolle et al., 2010), which focuses on differences between neighboring pixel values, the Gaussian prior applies smoothing filter globally across larger regions, reducing variations uniformly, including edges.
where 1 denotes the 1-th layer, LSS is the mean squared loss, and Les is the cosine similarity loss.
Visual Text Prior. Accurately reconstructing small-sized words (~1% of the full image) is highly challenging task due to their minimal contribution the overall gradient and their low visibility in the reconstructed image. To address this, we introduce a new text prior based on the Laplacian fil- ter (Paris et al., 20. The Laplacian filter is a second-order derivative filter that emphasizes regions of rapid intensity changes in an image, such as edges and fine-grained details. is particularly effective for extracting high- figh-frequency fea- tures, such as the boundaries of text and intricate patterns, which are crucial for reconstructing specific word details. The discrete form of the Laplacian filter measures the sum of curvature in both the horizontal and vertical directions, which is expressed as:
The TV prior is widely used in image reconstruction tasks to enforce smoothness while pre- serving important structures. Following prior works (Geip- ing et al., 2020; Hatamizadeh et al., 2022), we adopt the standard spatial TV formulation, which measures the varia- tion between neighboring pixels in the spatial dimensions of an image (i.e., width and height). For an image d'D with spatial dimensions (i. the spatial TV is defined as:
Total Variation Prior.
PED + 022
PDD
VaD((..))
(6)
05²
RivS((ID) = Σ 1DD - LDGiHl
02z and Jep are the second derivatives of the where 0i image intensity Z'D with respect to spatial coordinates and j. In practice, the Laplacian filter implemented using a
0j"²
(DD(ast2) - -DDI)) (9)
5

---

Gradient Inversion of Multimodal Models
on, our scheduler ensures that the optimization process is guided toward realistic reconstructions from the start. We adopt an exponential decay strategy for (t),, defined as (o()) = a exp(- where a is the initial weighting coeffi- cient and X is the scaling factor that determines the decay rate. This allows the contribution of auxiliary priors to diminish gradually as optimization progresses, letting the gradient matching loss take precedence in later stages for fine-tuning. This shift enables the priors shape the overall structure early on, while the gradient matching loss refines details as optimization converges.
20 75 100
125
150
175
200
225
250
Figure 3: Color palette where the RGB chamnels have iden- tical intensity (R-G=B). The palette ranges from black (R=G=B=0) to white (R=G=B=255), illustrating colors that vary solely in brightness without hue or saturation. This reflects the effect of the TV channel prior, which enforces RGB consistency and promotes grayscale reconstructions, aligning with the typical appearance of text and backgrounds in documents.
4. Evaluation
In addition to spatial smoothness, we introduce a channel TV prior, which operates on the color channels (i.e., red, green, and blue) at each pixel. The channel TV penalizes variations in color intensity within pixel across the RGB channels, helping suppress chromatic artifacts and reduc- ing color noise. For an image where the RGB channels are forced to have equal intensity (R=G=B), the resulting colors belong to the grayscale spectrum. This spectrum consists of shades that vary only in brightness, ranging from black (R=G=B=0) to white (R=G=B=1), without any hue or saturation, as shown in Figure 3. The channel TV prior leverages this property to encourage grayscale consistency across the reconstructed image, aligning with the nature of text reconstruction tasks, where content (e.g., text or back- ground) is predominantly black, white, or gray. The channel TV is defined as:
4.1. Evaluation Setup
Models. We use state-of-the-art DQA models, covering both OCR-free and OCR-based models: (i) Document Understanding Transformer (Donut) (Kim et al., 2022) an OCR-free model that employs transformer-based encoder-decoder architecture. Consists of a vision encoder (Swin Transformer (Liu et al., 2021)) and text decoder (BART (Lewis, 2019)). We use the base-size model with 156M parameters. (ii) Layoutl My33 (Huang et al., 2022) an OCR-based model and the latest version of the Lay- outLM family models (Xu et al., 2020a), which employs transformer-bus based architecture with visual patch embed- dings and incorporates textual input along with bounding box coordinates for spatial context (i.e., OCR). We use the base-size model, which consists of 133M parameters.
RIvcc(ED)= = Σ (.D()..)) - d'Daj..yy++
Datasets. We use the PFL-DocVQA (Tito al., 2024) dataset, designed perform Doc in FL environment, with the aim of exposing privacy leakage issues in realis- tic scenario. It comprises invoice document images and a set of question/answer pairs. The documents contain sensi- tive data of the invoice provider identity, including provider name, address, and other sensitive fields. For our experi- ments, we use a subset of the original dataset containing 395 documents. The subset includes 90 templates, each with approximately five distinct documents (the sensitive data differs between the documents of the same template).
(.fD(1..)G) - (D) (=)D(,,.))) daa.. (10)
Combining both spatial and channel priors, the overall TV prior expressed as:
RTv = RTV)(EN) + RTv.C(En)
(11)
By combining spatial and channel priors, the overall TV prior enforces both local smoothness across neighboring pixels and consistency in color channels. The inclusion of the channel TV prior ensures that reconstructed images adhere to the grayscale spectrum, effectively promoting natural and visually coherent reconstructions that align with the characteristics of text or background.
Metrics. In our evaluation, performance is measured with metrics commonly used in the visual gradient inversion do- main (Hatamizadeh et al., 2022; Lu et al., 2021; Geiping et al., 2020): (i) Peak Signal-to-Noise Ratio (PSNR) qual- ity of reconstructed images by quantifying the ratio between the maximum possible signal value and the distortion intro- duced by the reconstruction. (ii) Cosine similarity in the Fourier space (FFT2p) - similarity between the original and reconstructed images in the frequency domain. (iii) Mean Squared Error (MSE) average squared difference between corresponding pixels in the original and recon- structed images.
Priors Scheduler. Unlike prior work (Hatamizadeh et al., 2022) that delayed the activation of auxiliary priors until later stages of optimization to avoid suboptimal conver- gence, we propose a scheduler that assigns high weights to priors early in training. This approach stems from our obser- vation that auxiliary priors play a critical role in stabilizing the early stages of optimization, where the synthesized in- puts are far from convergence. By emphasizing priors early
6

---

Gradient Inversion of Multimodal Models
Table I: Main results. Evaluation of our attack and prior works on the Donut and LayoutL models. & indicates that lower values are better, and f indicates that higher values are better. Bold indicates superior results.
Donut
Lvy
Method
PSNR
FFT.0
MSE 4
Binary
Fuzz Ratio 1
PSNR f
FFT
MSE 4
Binary 1
Fuzz Ratio 1
Random
6.989
0.113
96.591
0%
7,041
0.124
105.169
0.9%
DLG (Zha nl., 2019)
6.984
0.110
96.574
0%
0.001
7.015
0.127
105. 185
1.1%
0
iDLG (Zhao et al., 2020)
6,984
0.110
96.574
0%
0.001
7.015
0.127
105.185
1.1%
0
Inverting Gradients (Geiping et al., 2020)
11.386
0.042
95.314
0%
10.596
0.044
98.281
0
APRIL (Lu et al., 2021)
13.652
0.027
85,329
5.8%
0.283
9.825
0.060
98.946
0.6%
0
GI-DQA (Ours)
24.194
0.003
60.403
70.1%
0.909
22.713
0.005
72.508
82.1%
0.681
Table 2: Effect of different loss components on the recon- struction quality. & indicates that lower values are better, and f indicates that higher values are better.
4.2.1. ATTACK EFFECTIVENESS
Table presents the results of our proposed attack in com- parison to prior works, including DLG (Zhu et al., 2019),
iDLG (Zhao et al., 2020), Inverting Gradients (Geiping et al.,
Loss PSNR 1
Reconstruction Metries
Component
FFT2D
MSB -.
Binary 1
Fuzz Ratio
2020), and APRIL (Lu et al., 2021). As shown, existing methods designed for unimodal models fail to reconstruct meaningful content when applied to multimodal architec- tures. DLG and iDLG attack performance is identical to that of a random baseline. Even APRIL, which is designed for transformer- based models, lacks the capability to capture the intricate interactions between visual and textual modal- ities, leading to incomplete or incoherent reconstructions. These methods struggle extract fine-grained document features, they are not designed to handle the fusion of structured text and image embeddings present in multimodal systems. In contrast, our proposed attack effectively exploits the unique properties of multimodal DQA models, leverag- ing both pre-fusion gradients (explained Section 5) and cross-modal dependencies to reconstruct sensitive document content with significantly higher fidelity. Furthermore, our approach is specifically designed to better fit the domain of visual text, ensuring more accurate recovery of structured textual elements that are critical in documents. For example, on the Donut model, our approach accurately reconstructs 70. 1 % of the set of words in all the documents combined, compared to 5.8% in APRIL, demonstrating its effective- ness in recovering fine-grained textual details. Examples are shown in Figure 4.
Land
17,872
0.012
75.008
42.6%
0.759
+ -RTv
22.555
0.006
63.823
61.0%
0.856
+ -RTss
24.540
0,003
58,063
70.2%
0.894
+ RRt
23.855
0.003
61.655
70.6%
0.913
Repa
24.194
0.003
60.403
70.1%
0.909
Phiaiie LSA Acall
Lgrd
+Rive +RIVS +RIE
+ Rgaa
Original
We also include metrics specifically designed for the DQA domain and visual text reconstruction. To achieve this, we use Trocr (Li et al., 2023), an OCR model to extract the re- constructed text, and measure: (i) Exact Match (Binary)- percentage of text segments that are perfectly reconstructed, reflecting the attack's 's ability to recover textual information without any character-level errors. (ii) Fuzz Ratio (FR) A similarity score based on character-level differences using fuzzy string matching, allowing for a more flexible evalu- ation of partial reconstructions where minor distortions or OCR errors may still preserve semantic meaning. This ap- proach evaluates the attack's effectiveness in an automated end-to-end scenario.
Implementation Details. The optimized pixels in the recon- structed document image are randomly initialized and up- dated using the Adam optimizer with an initial learning rate
4.2.2. ABLATION STUDIES
Priors Effect. We analyze the contribution of each train- ing loss term by evaluating its impact on reconstruction quality. Table 2 presents these results, providing quantita- tive comparisons alongside qualitative examples to illustrate their effectiveness. We observe that while optimizing Lyaa alone enables partial reconstruction, it struggles to eliminate the random noise introduced by the adversary's 's document initialization, achieving a binary match of 42.6%. Intro- ducing the channel-wise total variation prior Rive yields a substantial improvement, increasing the binary accuracy to 61.0% and the PSNR to 22.56 by promoting grayscale con- sistency and reducing chromatic noise. Adding spatial TV
of 2.0, applying exponential decay with a rate of À 0.999
over 5,000 iterations. The auxiliary loss terms (Equation 4) are weighted using the coefficients Ona 0.1, Ggam = 0.01,
and QTv = 0.1. These values were selected using the grid search approach over the values (.,01 0.00,,00,.0,,,.0 0.1, 1). op-
timizing for PSNR performance.
4.2. Results
Here, we present the results for our proposed attack. We report additional results in Appendix A. and provide exam- ples of reconstructed documents in Appendix A.3.
7

---

Gradient Inversion of Multimodal Models
Original
Censored
Inverting Gradients
APRIL
GI-DQA (Ours)
Property
WERQ-FM
Property
Property
Propery
Property
W/RO-FF
Invoce
13735086-2
Involce
Inoace
Invoice
Invoice
1G05222
June
Invcice Marth
Iroocc
Invoice Morth
Jure 2012
Pertod
0S/00/2-- 06155/2
Involce Period
Invace Perco
Invoice Period
Invoke Period
16:302P 06403022
Advertser
Franchor Andernon
Adventser
Advertser
Advertser
Adveriser
Product
Weeks before Pult
Prosuct
Product
Product
Product
[VVeeik heffow
Estimate
Advertiet
Adveriser
Abvetiast
Prienites USA Action
Friomtii USA Ro
Prionites USA Action
Dutaes
Estarate
Dares
Contact Dates
Contract Dates
Ec $253 2A
Contract Dates
(0300022
Prockcct
Prodkct
Ploduxt
TV
TV
POLIOWEN GRAHMMDOOWFL, (4075)
T RES hig aegenee
POLEGWEN GRANAIT DAGOWFL, (8673)
Screen Strategies Meda (238)
Screen Bivategica Media (288))
JONESJRACEEL
JONESRICCHE
POLITICAL POLITIALL (1962)
(h:
Policcll (1
Ganddeee
GRAHAM FOR GOVERNOR (1448)
GOVERVOR (1480)
Figure 4: Examples of zoomed-in sections of documents where private information is reconstructed using different gradient inversion techniques on the Donut model. Censored represents the document with redacted private data, which serves the adversary's s initial reference before optimization.
prior Riys further enhances smoothness across neighbor- ing pixels, leading to the best overall PSNR of 24.54, which indicates its strong contribution to local structural coherence. The text prior Rua slightly improves the binary match and fuzz ratio, sharpening edges and enhancing text legibility. The Gaussian prior Rgm helps suppress the remaining high- frequency noise, leading to an improved PSNR over Rea. stabilizing the reconstruction, and maintaining performance gains across all metrics.
Table 3: Impact of question and answer knowledge on doe- ument reconstruction ability. 4 indicates that lower values are better, and t indicates that higher values are better. "Original question and answer were replaced with random tokens.
Known Question
Reconstruction Metrics
Answer
PSNR 1
FFT22
MSE
Binary ?
Fuzz Ratio
x
5.237
0.196
87.212
0.8%
0
v
6.956
0.124
88.743
0.7%
0
x
5.240
0.195
87.207
1.0%
0
24.194
0.003
60.403
70.1%
0.909
24.399
0.002
59.677
71.9%
0.920
Question-Answer Effect. As discussed in Section 3.3, the effectiveness of document reconstruction depends on prior access to the question-answer (QA) pair. Table evaluates the attack's 's performance under different attacker knowledge scenarios: (i) black-box setting- both the question and an- swer unknown (random tokens are used). (ii) gray-bat setting only one of the question or answer is known. (iii) white-box setting- both the question and answer are ac- curately reconstructed and used in the attack. The results demonstrate that document reconstruction is infeasible with- out first recovering both the question and answer, even when one component is known. In contrast, when the question- answer pair is fully known, the attack significantly improves, enabling the successful reconstruction of private document pair high-
formation (PII), it typically reflects only small portion of the overall sensitive content. Additional PII fields, unmen- tioned in the QA, can still be inferred from the gradients (further discussed in the following section). Interestingly, our findings reveal that the semantic content of the question and answer is not critical to the attack's ' success. Instead, effective document reconstruction relies on the adversary having access the exact token identities and their posi- tions used by the client, regardless of their meaning. To demonstrate this, we replaced the original QA pair with ran- dom tokens of the same length and structure. The resulting reconstructions showed nearly identical quality across all metrics, as shown in Table 3 (bottom row). This indicates that gradient inversion exploits the precise token represen- tations encoded the model's gradients, rather than their semantic interpretation.
details. This reliance on the lights a unique characteristic of multimodal models, where the fusion of different modalities plays a crucial role in pre- serving and extracting information. Unlike unimodal attacks that rely solely on direct feature recovery, multimodal gradi- ent inversion depends on cross-modal interactions, making the reconstruction process inherently dependent on both textual and visual components. Although the answer in the QA pair may contain a piece of personally identifiable in-
Training Stage Effect. We conduct a systematic evalua- tion of the attack's 's effectiveness at different stages of model training to understand how the evolution of learned represen- tations influences reconstruction quality. Our findings reveal clear trend: as the model becomes more optimized, inver-
8

---

Gradient Inversion of Multimodal Models
Post-fusion gradients lack the fine- fine-grained details of the input document, indicating that future privacy-preserving strategies may focus on protecting pre-fusion layers rather than all layers indiscriminately.
13175068-3
1379900-32
06
Prudie Date
0009//2
Puoice Date
D666/2
Murih
Sime 1922
Morth
Jum 2012
Pered
DR/0022-
Dvoice Perted
Prepery
HEIRQ FM
L37 2
/
Bate
04
Pusic Date
96/922
Morth
Dune
Suonce Morth
Tune 2822
Parel
16900R 0666 08
WERO
Puice Pesce
WERN
95-9922 06/642
6. Defense
13P30066-3
05
Das
Dase
D6/9-22
Murch
Dane 2088
Morth
June 2022
In response to the threats posed by GI-DQA, we introduce Safe Template, a mitigation strategy that targets the adver- sary's dependence on publicly available document templates. Since organizations control these templates, they can proac- tively embed subtle, human-imperceptible perturbations into published versions to hinder reconstruction efforts. This de- fense is based on a key insight: gradient inversion attacks heavily depend on gradient matching, wherein the adversary aligns their reconstructions with the observed shared gradi- ents. By maximizing gradient divergence at the input level, Safe Template disrupts this alignment without altering the vi- sual appearance of the document. Unlike prior defenses that manipulate or obfuscate the gradients themselves (Li et al., 2022; Zhu et al., 2019), Safe Template perturbs the input images used to compute those gradients. We evaluate two strategies for introducing perturbations: (i) random noise, and (ii) adversarial noise crafted using projected gradient descent (PGD) (Madry al., 2017), with a custom objective Laefender = Cgras that explicitly maximizes gradient diver- gence. Both approaches prove to be effective in reducing the success of the attack. Random noise is effective for >> 255 .32
Insiis Panod
869002 066662
Puuce Pesce
85-9022 066562
/
Date
06/9222
Marth
Dune 2002
SHOKE Morth
June 8023
Penoo
D60692 060002
Proce Pettod
(6/3922 0609/22
13738008-2
22
06/6-2
Murth
Proce Morth
June 2002
Patid
D60099
066622
Phocc Pesko
56-9322 (669623
Figure 5: Cross-attention maps from the decoder for the input "06/05/22" split into tokens in fine-tuned model (left) and a randomly initialized model (right). The fine- tuned model exhibits sharply focused attention, while the randomly initialized model attends to broader regions, en- abling more recoverable information during inversion.
sion quality degrades significantly (full results can be found in Appendix A.1. This suggests that, early in training, the model encodes a broader and more spatially distributed set of features, as its cross-aftention tends to be more diffuse and less focused on specific regions. In contrast, as train- ing progresses, attention and token representations become increasingly concentrated on task-relevant areas, particu- larly those directly associated with the question and answer, thereby narrowing the gradient signal, as shown in Figure 5. This focused representation reduces the diversity of infor- mation available in the gradients, making it more difficult to reconstruct PII beyond the QA scope. This observation also helps explain why our attack remains effective even when QA tokens are replaced with random tokens, as long as the adversary has access to the same input tokens.
while PGD-based perturbations provide comparable defense even at e > 255 where e denotes the perturbation budget, which controls the intensity of modifications applied the images. Importantly, Safe Template is applied offline, im- poses computational overhead on the training process, and preserves full model utility, since benign clients train on clean, unmodified documents. This makes it a lightweight and practical defense for safeguarding multimodal FL sys- tems. For detailed results and analysis, see Appendix A.2.
5. Discussion
7. Conclusion
A key aspect of understanding privacy risks in multimodal models is identifying which gradients are most susceptible to inversion. In our analysis, we find that only a subset of the gradients meaningfully contribute to successful reconstruc- tion. Multimodal Dision-languaa models typically follow three stages: (i) extracting features from visual and tex- tual inputs, (ii) fusing these modalities, and (iii) generating the output. Our findings show that gradients from the pre- fusion stage, where the model still processes each modality independently, carry the most informative signal for recon- struction. In contrast, post-fusion gradients are significantly less revealing. In Donut, relying only on post-fusion gradi- ents makes reconstruction slower and less accurate, while in Mv3, reconstruction fails entirely. These findings suggest that multimodal fusion acts as an implicit safeguard.
In this work, we introduced GI-DQA, gradient inversion attack that reveals critical privacy vulnerabilities in multi- modal federated learning (FL) systems. By exploiting pre- fusion gradients and cross-modal interactions, GI-DQA suc- cessfully reconstructs private document content, exposing a significant threat overlooked by prior unimodal approaches, which fail in such settings. To address this risk, we proposed Safe Template, input-level defense that perturbs shared templates to disrupt gradient alignment, effectively mitigat- ing inversion without affecting model utility. Our findings underscore the need for privacy-aware architectural design and robust defensive strategies in multimodal FL deploy- ments. Future work should investigate adaptive defenses and training protocols that preserve both utility and privacy in cross-modal learning environments.

---

Gradient Inversion of Multimodal Models
Impact Statement
T., et al. An introduction total variation for image analysis. Theoretical foundations and numerical methods for sparse recovery, 9(663-34):227,, 2010.
This paper presents a study on gradient inversion attacks in FL. specifically targeting DQA models. Our work exposes significant privacy vulnerabilities in FL systems, particularly in scenarios where sensitive document data are involved, such as medical records, financial statements, and legal contracts.
Chu, H.-M., Geiping. J., Fowl, L. H.,, Goldblum, M., and Goldstein, T. Panning for gold in federated learning: Targeted text extraction under arbitrarily large-scale ag- gregation. In The Eleventh International Conference on Learning Representations, 2022.
Potential Ethical Concerns. Federated learning, despite its privacy-preservngg design, remains susceptible to gradi- ent inversion attacks that can reconstruct private training data, posing a risk individuals whose confidential infor- mation may be leaked. This is especially concerning for applications involving sensitive or personal documents. Al- thoughugh our research aims to improve the security of FL systems by exposing these vulnerabilities, it also carries the risk of adversarial actors misusing our findings to develop more effective attacks against real-world deployments.
Davis, B., Morse, B., Price, B., Tensmeyer, C.,, Wigington, C., and Morariu, V. End-to-end document recognition and understanding with dessurt. In European Conference on Computer Vision, pp. 280-296. Springer, 2022.
Deng, J., Wang, Y., Li, J.,, Shang, C., Liu, H., Rajasekaran, S., and Ding, C. Tag: Gradient attack on transformer- based language models, arXiv preprint arXiv:2103.06819, 2021.
Mitigation and Responsible Use. Our work underscores the urgent need for robust defenses, such as gradient ob- fuscation techniques, differential privacy mechanisms, and secure aggregation strategies, to mitigate the identified risks. We also present a defense mechanism to mitigate the risks posed by such attacks. We encourage the research commu- nity to use our findings to strengthen methods in FL rather than to exploit them maliciously. Fur- thermore, adhere responsible disclosure practices and have framed our research in way that prioritizes defense- oriented contributions.
Du, J., Hu, J., Wang. Z., Sun, P., Gong, N. Z., and Ren, K. Sok: Gradient leakage in federated learning, arXiv preprint 2XXv:24044004003, 2024.
Fan, L., Ng, K. W., Ju, C., Zhang, T., Liu, C., Chan, C. S., and Yang. Q. Rethinking privacy preserving deep learn- ing: How to evaluate and thwart privacy attacks. Feder- ated Learning: Privacy and Incentive, pp. 32-50, 2020.
Fowl, L., Geiping, J., Czaja, W., Goldblum, M., and Gold- stein, T. Robbing the fed: Directly obtaining private data in federated learning with modified models. arXiv preprint arXiv:2110.1300 2021.
Broader Societal Implications. By demonstrating vulnera- bilities in multimodal FL systems, we contribute to ongoing efforts enhance the security and privacy of decentral- ized machine learning frameworks. Furthermore, our study informs policymakers, developers, and researchers about potential risks in privacy-sensitive Al applications, helping to shape future regulatory and technical safeguards. In con- clusion, this research seeks to advance the field of machine learning by improving the understanding of privacy risks in FL., with the ultimate goal of fostering more secure and privacy-aware AI systems.
Fowl, L., Geiping, J., Reich, S., Wen, Y., Czaja, W., Gold- blum, M., and Goldstein, T. Decepticons: Corrupted transformers breach privacy in federated learning for lan- guage models. arXiv preprint arXiv:2201.1267. 2022.
Geiping. J., Bauermeister, H., Dröge, H., and Moeller, M. Inverting gradients-how easy is it to break privacy in federated learning? Advances in neural information processing systems, 33: 13:16977-16947, 2020.
Gupta, S., Huang, Y., Zhong, Z., Gao, T., Li, K., and Chen, D. Recovering private text federated learning of lan- guage models. Advances in neural information process- ing systems, 35:8130 1113333-3333 8143, 2022.
References
Balunovic, M., Dimitrov, D., Jovanović, N., and Vechev, M. Lamp: Extracting text from gradients with language model priors. Advances in Neural Information Processing Systems, 35:7641-7654, 2022.
Hatamizadeh, A., Yin, H., Roth, R., Li, W., Kautz, J., Xu, D., and Molchanov, P. Gradvit: Gradient inversion of vision transformers. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10021-10030, 2022.
Banabilah, S., Aloqaily, M., Alsayed, E., Malik, N., and Jararweh, Y. Federated learning review: Fundamentals, enabling technologies, and future applications. Informa- tion processing & management, 103061, 2022.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016
Chambolle, A., Caselles, V., Cremers, D., Novaga, M., Pock,
10

---

Gradient Inversion of Multimodal Models
Huang. Y., Lv, T., Cui, L.,, Lu, Y., and Wei, F. Layoutmmv:: Pre-training for document ai with unified text and image masking, In Proceedings of the 30th ACM International 2022.
Nguyen, K. and Karatzas, D. Federated document visual question answering: A pilot study. In International Con- ference Document Analysis and Recognition, pp. 146- 163. Springer. 2024.
Conference on Multimedia, pp.
Kim, G., Hong, T., Yim, M.. Nam, J., Park, J., Yim, J., Hwang, W., Yun, S., Han, D., and Park, S. Ocr-free docu- ment understanding transformer. In European Conference on Computer Vision, pp. 198--17.50 Springer, 2022.
Paris, S., Hasinoff, S. W., and Kautz, J. Local laplacian filters: Edge-aware image processing with a laplacian pyramid. ACM Trans. Graph., 30(4):68, 2011.
Pasqunni, D., Francati, D., and Ateniese, G. Eluding secure aggregation in federated learning via model inconsistency. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pp. 2429-2443, 2022.
Lee, K.. Joshi, M., Ture, I. R., Hu, H., Liu, F., Eisensch- los, J. M., Khandelwal, U., Shaw, P., Chang, M.-W., and Toutanova, K. Pix2struct: Screenshot parsing pretraining for visual language understanding. In Inter- national Conference on Machine Learning, pp. 18893- 18912. PMLR, 2023.
Paulik, M., Seigel, M., Mason, H., Telaar, D., Kluivers, J., van Dalen, R., Lau, C. W., Carlson, L., Granqvist, F. Vandevelde, C., et al. Federated evaluation and tuning for on-device personalization: System design & applications.
Lewis, M. Bart: Denoising eequence-to-sequenc pre- training for natural language generation, translation, and comprehension. arXiv preprint
2019.
arXiv preprint arXiv:2102.08555 2021.
Li, M., Lv, T., Chen, J., Cui, L., Lu, Y., Florencio, D., Zhang, C., Li, Z.., and Wei, F. Troer: Transformer-based optical character recognition with pre-trained models. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 37, pp. 10094-13102, 2023.
Petrov, I,, Dimitrov, D. I., Baader, M., Müller, M. N., and
Vechev, M. Dager: Exact gradient inversion for large lan- guage models. arXiv preprini arXiv:2405. 2024.
Powalski, R., Borchmann, E., Jurkiewicz, D., Dwojak, T., Pietruszka, M., and Palka, G. Going full-tilt boogie on document understanding with cext-image-layout trans- former. In Document Analysis and Recognition-ICIDA 2021: 16th International Conference, Lausanne, Switzer- land, September 5-10, 2021, Proceedings, Part II 16, pp. 732-747. Springer, 2021.
Li, Z., Zhang, J., Liu, L., and Liu, J. Auditing privacy defenses in federated learning via generative gradient leakage. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10132- 10142, 2022.
Liu, X., Cai, S., He, R., and Yuan, J. Mutual gradient inversion: Unveiling privacy risks of federated learning on multi-modal signals. IEEE Signal Processing Letters, 2024.
Shi, Y. Song, H., and Xu, J. Responsible and effective federated learning in financial services: A comprehensive survey. In 2023 62nd IEEE Conference on Decision and Control (CDC), pp. 4229 4236. IEEE, 2023.
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022, 2021.
Tang, Z., Yang, Z., Wang, G., Fang, Y., Liu, Y., Zhu, C., Zeng, M., Zhang, C., and Bansal, M. Unifying vision, text, and layout for universal document processing. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 19254-19264, 2023,
Lu, J. Zhang, X. S., Zhao, T., He, X., and Cheng, J. April: finding the achilles' heel on privacy for vision transform- ers. in 2022 ieee. In CVF Conference on Computer Vi- sion and Puttern Recognition (CVPR), pp. 10041-10050, 2021.
Thrasher, J., Devkota, A., Siwakotai, P., Chivukula, R., Poudel, P., Hu, C., Bhattarai, B., and Gyawali, P. Multi- modal federated learning in healthcare: a review. arXiv preprint arxiv: 20.006650 2023.
Tito, R., Nguyen, K., Tobaben, M., Kerkouche, R., Souibgui, M. A., Jung, K., Jalkö, J., D' Andecy, V. P., Joseph, A., Kang, I.,, et al. Privacy-aware document visual question answering. In International Conference Document Analysis and Recognition, pp. 199-218. Springer, 2024.
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. arXiv preprint 2017.
McMahan, B., Moore, E., Ramage, D., Hampson, S., and Arcas, B. A. Communication-efficienn learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282, PMLR, 2017.
Wang. X., Chen, G., Qian, G., Gao, P., Wei, X.-Y., Wang, Y., Tian, Y., and Gao, W. Large-scale multi-modal pre- trained models: A comprehensive survey. Machine Intel- ligence Research, 20(4):447-45) 482, 2023.

---

Gradient Inversion of Multimodal Models
Wen, Y., Geiping, J., Fowl, L., Goldblum, M., and Gold- stein, T. Fishing for user data in large-batch feder- ated learning via gradient magnification. arXiv preprint arXiv:2202.00580, 2022.
Xu, Y., Li, M., Cui, L., Huang, S., Wei, F., and Zhou, M. Layoutlm: Pre-training of text and layout for document image understanding, In Proceedings of the 26th ACM SIGKDD international conference on knowledge discov- ery & data mining, pp. 1192-1200, 2020a.
Xu, Y., Xu, Y., Lv, T., Cui, L.,, Wei, F., Wang, G., Lu, Y., Florencio, D., Zhang, C., Che, W., et al. Layoutlmv2: Multi-modal pre-training for visually-rich document un- derstanding, arXiv preprint arXiv:2012.144: 14740, 2020b.
Yang, T., Andrew, G., Eichner, H., Sun, H., Li, W. Kong, N., Ramage, D., and Beaufays, F. Applied federated learning: Improving google keyboard query suggestions. 2018.
arXiv preprint arXiv:
Yin, H., Mallya, A., Vahdat, A., Alvarez, M., Kautz, J., and Molchanov, P. See through gradients: Image batch recovery via gradinversion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16337-16346, 2021.
Zhao, B., Mopuri, K. R., and Bilen, H. idlg: Improved deep leakage from gradients. arXiv preprint arXiv:2001.02610, 2020.
Zhu, J. and Blaschko, M. R-gap: Recursive gradient attack privacy. arXiv preprint arXi::2010.07733, 2020.
Zhu, L., Liu, Z., and Han, S. Deep leakage from gradients. Advances in neural information processing systems, 32, 2019.
12

---

Gradient Inversion of Multimodal Models
A. Appendix
A.I. Additional Results
A.1.1. ATTACK PERFORMANCE ON BATCHES
We also examine the effect of different batch sizes on the quality of reconstruction, as reflects a more complex scenario which the shared gradients are averaged over the entire batch. In our experi- ments, we demonstrate the ability reconstruct single document when the shared gradients Table 4 presents the results of the reconstruction using batch sizes of two, four, and six. Our experiments demonstrate that our attack remains effective when gradients are computed from up to two sam- ples, successfully reconstructing a single document these cases. However, with larger batch sizes, the quality of reconstruction di- minishes. This effect stems from the nature of text-based gradients. which tend to be more sparser and localized compared to visual features. When gradients are averaged over larger batches, the seeument-speciffe signal becomes more diffused, making it harder to isolate fine fine-grained textual details. Nevertheless, despite this increased complexity, our approach remains capable of extracting meaningful information, demonstrating its robustness in multi-sample scenarios.
Table 4: Effect of different batch sizes on the re- construction quality. indicates that lower values are better, and f indicates that higher values are better.
computed using various batch sizes.
Batch Size
Reconstruction Metries
PSNR
FF +
MSE 4
Binary
Puzz Ratio T
49.6%
0.452
7.155
0.132
90.461
0.7%
6
8.215
0,094
94.001
0.6%
A.1.2. TRAINING STAGE EFFECT
As noted in Section 4.2.2, we conducted comprehensive evaluation of the effectiveness of the attack at various stages of model training. As shown in Figure 6, the ability to reconstruct document pixels progressively degrades as the model converges. For example, at initialization (iteration 0), approximately 70.1 % of words are perfectly reconstructed, whereas by iteration 30, the success rate drops below 10%.
Attack Success (Binary) vs. Training Iteration
Sharr 191
10
Thaining
Figure 6: Attack success rate (Binary accuracy) as a function of training iteration. The results show clear downward trend in reconstruction performance as the model converges.
A.1.3. EFFECT OF PRIOR SCHEDULING STRATEGY
To assess the impact of the prior scheduling strategy on reconstruction quality, we compare our proposed approach with two alternatives: (i) the delayed scheduling strategy of GradViT (Hatamizadeh et al., 2022), and (ii) a baseline without any scheduling mechanism, where prior weights are kept constant throughout the optimization. GradViT delays the introduction of auxiliary priors until the later stages, allowing the model initially optimize solely based on the gradient matching loss. In contrast, our scheduler assigns higher weights to auxiliary priors during early iterations and gradually reduces their influence via an exponential decay schedule. This design aims to stabilize the early stages of optimization, especially in settings where the template reduces the gradient signal in sensitive regions, by encouraging visually coherent structures from the outset.
Table 5 summarizes the results. Our scheduler is the only one to produce viable reconstructions, achieving strong visual and
13

---

Gradient Inversion of Multimodal Models
semantic fidelity across all metrics. In contrast, both GradViT and the constant-weight baseline fail to guide the optimization meaningfully: GradViT's delayed prior activation leads to noisy and fragmented outputs, while constant weighting results in low-fidelity reconstructions that collapse semantically (i.e., near-zero binary accuracy and fuzz ratio). These findings highlight the importance of strong early guidance from auxiliary priors when gradient signals are weak or ambiguous.
Table 5: Comparison of prior scheduling strategies. Our early-weighted scheduler leads significantly improved recon- structions across visual and semantic metrics.
Scheduler
PSNR
FFT
MSE +
Binary t
Fuzz Ratio
None
10.854
0.048
94.781
5.1%
0
GradVIT
8.871
0.080
79.440
0.1%
0.042
Ours
24.194
0.003
60.403
70.1%
0.909
A. 1.4. EFFECTIVENESS OF INDIVIDUAL PRIORS
To better understand the role of each auxiliary prior our reconstruction objective, we evaluate them independently rather than cumulatively (Section 4.2.2). Table 6 presents the results of optimizing each loss term in isolation. The Laplacian prior (Rtxt) and the spatial TV prior (RTV-S) achieve the strongest overall performance, significantly improving both perceptual quality (highest PSNR) and textual fidelity (lowest fuzz ratio, highest binary match). The channel TV prior (Rrvec) also performs well, particularly in maintaining semantic consistency. In contrast, the Gaussian prior (Rgau) achieves relatively modest performance when used alone. However, this does not reflect its true value; when used in combination with other priors (as shown in our main ablation), plays a critical role in stabilizing optimization and suppressing noise. Finally, attacks without any regularization (Cgrad only) exhibit degraded results across all metrics, highlighting the necessity of auxiliary priors for successful reconstruction.
Table 6: Performance of individual priors based on reconstruction metrics.
Loss PSNR 1
FFT2D +
MSE +
Binary Test
Fuzz Ratio
Component
Egdd
17.872
0.012
75.008
42.6%
0.759
Rgau
18.246
0.010
74.701
45.1%
0.771
Rixe
21.182
0.005
71.783
66.6%
0.896
Rive
20.519
0.007
71.501
60.8%
0.874
Ryyss
20.508
0.006
72.915
64.2%
0.888
All Combined
24.194
0.003
60.403
70.1%
0.909
A.2. Defense
We propose Safe Template, practical defense mechanism designed to mitigate gradient inversion attacks on multimodal models, particularly DQA models within FL setups. This defense is motivated by a key vulnerability exploited in our attack: the adversary's 's access to publicly available document templates. These templates are frequently shared by organizations for standardization and accessibility purposes, and their consistent structure enables adversaries to significantly narrow the search space during reconstruction. The success of a gradient inversion attack depends on the similarity between gradients computed on the attacker's reference template and those generated by the actual client. Safe Template disrupts this similarity by embedding small, targeted perturbations into publicly released templates, thereby diminishing their usefulness for adversarial inversion.
1. DEFENSE OVERVIEW
Safe Template introduces perturbations into the document template that maximize the distance between the gradients computed on clean versus perturbed inputs. These perturbations are generated offline, prior to FL training and do not interfere with client utility. The benign client continues to use the unperturbed document during FL training, ensuring model accuracy and gradient integrity.
14

---

Gradient Inversion of Multimodal Models
Algorithm Safe Template
Input: Model fa. visual document dD, question zQ, answer yanss perturbation budget 6, norm p, step size a Output: Perturbed Template
1: 8 =0
> Step 1:
2: yans = fe( xQ)
3: feleam &mmudeel Wanss Vans)
4: VOetean 00
Step 2:
5: repeat
6:
DD = DD + 8
per
7:
Uper = Jo (1' per xQ))
8:
Lper museel Oper, Yans)
Vpee U.
9:
Step 3:
10:
Ldel Earad D see Section 3.3.1
11: ddet aCaa
12:
II
(6
sign( sign(Vode)) V(ae))
||
13:
until Convergence
14:
Return
The method consists of three steps, outlined in Algorithm 1, which can be summarized as follows:
Step 1: Clean Gradient Computation. Given a document-question pair (æD, IQ). the model performs a forward pass and predicts an answer ÿ. The loss is computed with respect to the ground-truth answer y, and the gradients Vietann are obtained by backpropagation. This step performed once per template.
Step 2: Adversarial Gradient Computation. A perturbation ô is added the document, yielding (æD + 8,x)). The model performs forward pass and backpropagation to compute pp the gradients the perturbed template.
Step 3: Perturbation Update. The perturbation 5 is optimized to maximize the gradient discrepancy, using the defender's loss Laet Liraad Gradients with respect to Lueeender are computed, and 5 is updated via Projected Gradient Descent (PGD) to remain within an Ep norm ball of radius e:
8
II Se
(8, + a sign (Ver))
where the step size and I[ denotes projection under norm constraint || until convergence.
!|. Steps and are repeated iteratively
A.2.2. KEY PROPERTIES
Safe Template provides several practical advantages:
Offline and Efficient. The perturbation generation process is performed once per template and does interfere with FL training, making computationally efficient.
Accuracy Preservation. Since benign clients use the original, unperturbed documents during training, model accuracy and utility are fully preserved. Gradients transmitted the server remain unaffected.
Scalable and Deployable. Organizations embed perturbations directly into the public template files they distribute, requiring no client-side or server-side modification.
A.2.3. RANDOM VS. ADVERSARIAL PERTURBATIONS
To evaluate the robustness of Safeeaat we explore two strategies for perturbation generation: (i) adversarially crafted perturbations, and (ii) random noise; Both approaches modify document pixels directly in a way that remains imperceptible
15

---

Gradient Inversion of Multimodal Models
the human eye, preserving the visual integrity of the template. To investigate the trade-off between privacy and document fidelity, experiment with varying perturbation budgets, denoted by 6, which control the maximum allowable pixel modification. Table 7 reports the effectiveness of both strategies on the Donut model.
The results demonstrate that both methods substantially reduce the success of the attack. Random perturbations become effective at > 255 lowering the reconstruction success rate to just 0.3%. In contrast, adversarial perturbations achieve similar protection at a much smaller budget (e
32
_h highlighting that even minimal, strategically crafted modifications
are sufficient to disrupt gradient inversion.
Table 7: + indicates that lower values are better, and f indicates that higher values are better. Bold indicates superior results.
Reconstruction Metries
Strategy
PSNR 4
FFT
MSE 1
Binary
Fuzz Ratio
No Defense
24. 194
0.003
60.403
70.1%
0.909
12/255
13.905
0.024
80.876
11.6%
0.064
Random
32/255
9.036
0,073
84.946
0.3%
0
8/255
12.027
0.037
84.550
0.25%
0.01
PGD
12/255
10.210
0.056
86.014
0.4%
0.002
16/255
9.004
0.075
86.961
0.4%
0.001
A.2.4. FL WITH LOCAL DIFFERENTIAL PRIVACY (LDP)
To assess the effectiveness of standard FI. defenses, we conducted complementary experiment simulating typical LDP mechanism. Specifically, we added Gaussian noise directly to the shared gradients, a widely adopted strategy for per-client privacy preservation in FL. As shown in Table 8, increasing the noise scale a leads to gradual reduction in attack success. At lower noise levels (a = 1 10-9 and 1 10-8), the gradients still retain enough signal for partial reconstruction, with binary match rates around 44 46%. At higher noise scale (a = 1 10-"), the attack is largely neutralized, although this level of perturbation may adversely affect model convergence and utility in practice. These findings highlight the limitations of gradient-level defenses: stronger privacy comes at the cost of model performance. In contrast, Safe Template operates entirely at the input level without modifying gradients, preserving training stability while substantially degrading the adversary's s ability to extract sensitive information.
Table 8: Effectiveness of LDP under different noise scales. Higher a implics stronger noise added to shared gradients.
PSNR J.
FFT 1
MSE 1
Binary
Fuzz Ratio +
Defense
24.194
0.003
60.403
70.1%
0,909
T-70 70
19.566
-..07
74.316"
75.9%.
0.786
1- 10-
17.805
0.010
77,412
0.825 44.0%
1 10-7
7.013
0.119
82.807
0%
0,001
A.3. Visualizations
In Figure 7, we visualize full-sized reconstructed documents where private information is recovered.
16

---

Gradient Inversion of Multimodal Models
Original
Censored
GI-DQA (Ours)
INVOICE
INOICC
INVOICE
820 JAMES
82Q
NONOEN
JAMES
CAM
FO
FO PLUS
FOX
PLUS
PLUS
Weigen
T WEVI
WEVI
WONT
Seeect fales N62000
DL IVVICE
CONTRACT
CONTRACT
CONTRACT
ObC
abc
ObG
Figure 7: Examples of documents where private information is reconstructed using our proposed attack. Censored represents the document with redacted private data, which serves the adversary's 's initial reference before optimization.
17


Reviews:
[{"review_id": "fR5fTx6fie", "reviewer": "Reviewer_fEpi", "summary": "This paper presents a novel approach to gradient inversion attacks (GI-DQA) on multi-modal models specifically targeting extraction of textual information in Document Question Answering (DQA) tasks. The authors demonstrate why gradient inversion attacks designed for targeting unimodal models trained for image classification tasks would be a subpar choice to attack multi-modal models like those used for DQA. They clarify their rationale behind GI-DQA and how it is a more effective choice to attack pre-fusion stage gradients in DQA models. Further, they propose defences that could help mitigate GI-DQA using document-level perturbations.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his time, effort, and valuable suggestions.\n\nQ1. \"..confidence intervals..\"\n\nA1. Our original submission reported mean values, as we observed low variance across runs, confirming that our method is stable and consistently outperforms baseline methods.\nWe will include the standard deviation values in the final version of the paper and provide additional details in the appendix to enhance transparency and reproducibility.\n\nQ2. \"..defences..utility..\"\n\nA2. Our proposed defense strategy applies perturbations to the input templates outside the FL training procedure. These perturbations are added to publicly available or previously observed templates and are used solely by the attacker as priors during the inversion process—not by the participating clients during training (which use \"clean\" documents).\nAs a result, the actual training inputs, local computations, and shared gradients remain completely untouched, and the federated learning process proceeds as usual. This ensures that the utility of the multi-modal models is fully preserved, unlike defenses that modify gradients directly (e.g., via differential privacy), which often degrade model performance [1].\nWe will update the paper to clarify this distinction and emphasize the utility-preserving nature of our defense.\n\nQ3. \"..performance of GI-DQA..\"\n\nA3. We thank the reviewer for this thoughtful observation. To clarify, overfitting is not a factor in our experiments, as all models—both OCR-based (e.g., LayoutLMv3) and OCR-free (e.g., Donut)—are evaluated using randomly initialized weights (see Reviewer fLco, Answer 5). This ensures that performance differences arise from architectural properties, not training dynamics.\nWe hypothesize that the variation in GI-DQA performance stems from differences in the size and structure of the visual backbones. LayoutLMv3 uses a smaller image encoder, leading to lower representational complexity and making gradient inversion more tractable. Donut, in contrast, uses a larger and more expressive visual backbone, which produces more abstract and distributed features—making inversion harder without strong priors.\n\nWe will revise the manuscript to clarify this architectural difference and its impact on attack performance.\n\nQ4. \"Differential Privacy (DP)..\"\n\nA4. We fully agree that Differential Privacy (DP) is a well-established and rigorous approach for protecting client data in federated learning.\nWhile defense design is not the main focus of our work, we propose a lightweight input-level method that operates entirely outside the FL training loop, leaving shared gradients untouched. This makes it a practical alternative when DP integration may be too costly or degrade utility.\nTo provide a point of comparison, we evaluated a DP-style defense by adding Gaussian noise to the shared gradients. Results are shown below:\n|Experiment|PSNR|FFT2D|MSE|Binary|Fuzz Ratio|\n|-|-|-|-|-|-|\n|DP|5.578|0.176|86.743|10.9%|0.000|\n|Ours|9.004|0.075|86.961|0.4%|0.001|0.001|\n\nWhile DP yields lower PSNR (better in terms of defense), our method more effectively disrupts text recovery, as shown by binary accuracy. This demonstrates the complementary nature of our defense, which focuses on degrading machine-readability, even if some visual structure remains.\nWe will include this discussion and table in the final version of the paper.\n\nQ5. \"..use GradViT..\"\n\nA5. We originally intended to include GradViT as a baseline in Table 1, given its relevance to gradient inversion in vision models. However, the authors of GradViT did not release their code, and despite reaching out to them multiple times, we unfortunately did not receive a response. Given the complexity of accurately reproducing their method—particularly in the context of multi-modal models—we decided against including potentially inaccurate or incomplete re-implementations. We would be happy to include GradViT as a baseline in future versions if their code becomes available.\n\nQ6. \"..Gaussian.. in addition..Laplacian..\"\n\nA6. While Table 2 may suggest the Gaussian prior is less impactful than Laplacian, we include both as they address complementary aspects of reconstruction. The Gaussian prior promotes smoothness and stabilizes early optimization by reducing noise, while the Laplacian prior sharpens edges and preserves fine details—crucial for small text recovery.\nSee Reviewer xopj, Answer 1 for the ablation of individual priors and Answer 4 for insights into the selection of priors.\n\n[1] Cummings, R., Desfontaines, D., Evans, D., Geambasu, R., Huang, Y., Jagielski, M., ... & Zhang, W. Advancing Differential Privacy: Where We Are Now and Future Directions for Real-World Deployment. Harvard Data Science Review, 6 (1), jan 16 2024.", "claims_and_evidence": "Yes, the authors thoroughly clarify their choice of the components of the loss term used for designing GI-DQA. I found none of the claims problematic. Furthermore, they clearly state the adversary's capabilities and the federated learning setting wherein the attack would be useful.", "methods_and_evaluation_criteria": "Given that there has been no prior work targeting multi-modal models using gradient inversion attacks, the authors chose to use prior works designed to reconstruct images from unimodal models as the baseline in Table 1. My only issue was a lack of comparison with the more recent GradViT attack by Hatamizadeh et al. which they refer to in subsection 2.1 but do not include in Table 1.", "theoretical_claims": "The paper contains no theoretical claims/ proofs to review.", "experimental_designs_or_analyses": "I want to highlight the following issues with the experimental design:\n1. No confidence intervals were reported in the paper. This makes it hard to judge how stable the performance of the attack is with multiple repeats.\n2. In line 419, Section 6, the authors present their proposed defences as less harmful to the utility of the multi-modal models. They do not follow this statement with numbers to prove their assertion.", "relation_to_broader_scientific_literature": "Prior works on gradient inversion attacks in federated learning frameworks, such as APRIL [1] seem to focus on recovering images from unimodal vision models meant for image classification tasks. Additionally, they are optimal for recovering images depicting a single object. With this work, the authors demonstrate the ineffectiveness of such approaches to recover private textual information in DQA tasks. They further develop an attack that could be much more useful for targeting DQA models", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\n1. The attack is agnostic of the nature of DQA models. Works well with OCR and non-OCR models. \n2. The attack combines direction and magnitude matching for computing gradient matching loss.\n3. Incorporates prior knowledge in the loss term to enable effective text data reconstruction.\n4. Proves effectiveness of their approach against prior attacks which were designed for unimodal models highlighting the superiority of the proposed attack.\n5. Highlights the implicit privacy safeguard offered by the fusion of text-based and visual features in the training of multi-modal models which makes post-fusion gradients less susceptible to GI-DQA.\n\nWeaknesses:\n\n1. The performance of GI-DQA appears to vary across OCR and non-OCR models. One possibility is that this could be simply because of the difference in size of these models. Larger models are often more susceptible to overfitting which increases their vulnerability to privacy attacks but this could also stem from the difference in the nature of the two types of DQA models. I would urge the authors to clarify the factors which affect the performance of GI-DQA against OCR and non-OCR models.\n\n2. No confidence intervals reported in the paper. This makes it hard to judge how stable the performance of the attack is with multiple repeats.\n\n3. Differential Privacy (DP) [3] is considered the gold standard for protecting clients' data via gradient perturbation. McMahan et al. [4] even introduced a variant of DP for federated learning. Author's do not compare their proposed defences against a baseline where the clients' gradients are protected by DP.", "other_comments_or_suggestions": "None", "questions_for_authors": "Questions:\n\n1. May I know why the authors chose not to use GradViT [2] as one of the baselines in Table 1? \n\n2. From Table 2, it is unclear that using the Gaussian prior in addition to the Laplacian filter-based prior is contributing significantly to improving the reconstruction. What then justifies the inclusion of the former term in the loss computation?\n\n3. A lack of comparison with DP makes it hard for me to understand whether these defences offer effective protection while preserving the utility of the models. Could the authors clarify their stance on this?\n\nP.S. If the authors are able to address the questions raised above, I am amenable to raising my score.\n\n[1] Jiahao Lu, Xi Sheryl Zhang, Tianli Zhao, Xiangyu He, Jian Cheng. APRIL: Finding the Achilles' Heel on Privacy for Vision Transformers. CVPR 2021. https://arxiv.org/abs/2112.14087\n\n[2] Ali Hatamizadeh, Hongxu Yin, Holger Roth, Wenqi Li, Jan Kautz, Daguang Xu, Pavlo Molchanov. GradViT: Gradient Inversion of Vision Transformers. CVPR 2022. https://arxiv.org/abs/2203.11894\n\n[3] Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep Learning with Differential Privacy. ACM CCS '16. https://doi.org/10.1145/2976749.2978318\n\n[4] H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. ICLR 2018. https://openreview.net/forum?id=BJ0hF1Z0b.\n\nP.P.S. The authors have addressed concerns/ questions raised by me adequately (with proof). Accordingly, I am raising my score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IxRHoYegy0", "reviewer": "Reviewer_fLco", "summary": "The paper proposes a gradient inversion attack targeting multi-modal DQA models in Federated Learning (FL) setups: GI-DQA. \n\nIn DQA models, the input consists of both a document and its corresponding question, while the output is the target answer. GI-DQA first employs existing methods to reconstruct question-answer pairs and then aims to reconstruct the input document. The experiments are conducted using two DQA models, Donut and LayoutLMv3, with the PFL-DocVQA dataset. The results demonstrate that GI-DQA outperforms other methods across various evaluation metrics.\n\n## update after rebuttal\n\nI increased the grading based on the rebuttal", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his time, effort, and valuable suggestions.\n\nQ1. \"found [r1]..\"\n\nA1. We thank the reviewer for pointing out this work. [r1] explores gradient inversion in multi-modal FL using synthetic image-text pairs processed by separate, modality-specific models trained independently, with no shared representations. The attack is coordinated via mutual knowledge distillation.\nWhile valuable, we believe the use of the term “multi-modal” in [r1] is somewhat misleading, as it refers to parallel models that only interact during inversion—unlike standard multi-modal architectures that jointly fuse modalities (e.g., via cross-attention).\nOur work focuses on such deeply fused, jointly trained models like those used in DQA. Nonetheless, we appreciate this connection and will include a discussion of [r1] in the related work section.\n\nQ2. \"theoretical analysis..\"\n\nA2. We would like to clarify that our work follows a well-established line of research on gradient inversion, which has primarily focused on empirical demonstration of vulnerabilities rather than formal theoretical analysis. Foundational works such as DLG (Kim et al., 2019), iDLG (Zhao et al., 2020), and GradViT (Hatamizadeh et al., 2022) have aimed to develop practical attack strategies and assess reconstruction quality under various conditions.\nGiven the complexity and high dimensionality of modern neural networks—especially in multi-modal setups like DQA—formal guarantees remain an open challenge. As in prior work, we rely on rigorous experimental evaluation across architectures and settings.\nOur work extends gradient inversion to multi-modal architectures, with ablations isolating each design component’s effect.\n\nQ3. \"FL setup..\"\n\nA3. We appreciate the comment and clarify that Federated Learning is a training paradigm, not a model type—it can be applied to any architecture, including Donut and LayoutLMv3.\nOur setup simulates gradients computed locally by a client on private data, which are then forwarded to a central server—where the adversary is assumed to reside. This is consistent with prior gradient inversion works and reflects a server-side threat model.\nSince our focus is on leakage from a single client, the total number of clients does not affect the core attack. We will revise the paper to clarify this setup.\n\nQ4. \"Priors Effect..\"\n\nA4.  Please see Reviewer xopj, Answer 1.\n\nQ5. \"iteration t..\"\n\nA5. We thank the reviewer for this interesting and important question. In our experiments, the attack was performed at iteration t=0, using the model’s initial random weights.\nWe also ran the attack at later training stages and found it to be significantly less effective. We hypothesize that in well-trained models, attention and token representations become sharply focused on regions relevant to the question and answer, which narrows the gradient signal. In contrast, early in training, encode more diverse and spatially distributed information, making reconstruction easier.\nTo support this, we include a [visualization](https://drive.google.com/drive/folders/1NxT7gBMSH5RQb9V1bEAuTT2AvXPqhex1?usp=drive_link) of cross-attention maps for both randomly initialized and fine-tuned models, showing how attention becomes increasingly narrow and localized when the model is finetuned.\n[Here](https://drive.google.com/drive/folders/16way4IQb3i2Thc_2K9fhAImCMhEyeFjk?usp=drive_link), we present a systematic analysis of attack success across training iterations, confirming that inversion becomes progressively more difficult as the model converges.\n\nQ6. \"QA Effect..\"\n\nA6. Thank you for raising this point. As shown in Section 4.2.2, gradient inversion fails without access to both the question and answer tokens, highlighting the need for semantic grounding.\nHowever, when we say the specific content is irrelevant, we mean the attack depends on the token identities and positions, not their semantic meaning. To demonstrate this, we replaced the original QA with random tokens of the same length and structure. The reconstruction quality remained nearly identical:\n|Experiment|PSNR|FFT2D|MSE|Binary|Fuzz Ratio|\n|-|-|-|-|-|-|\n|OriginalQA|18.519|0.008|79.853|0.419|0.688|\n|RandomQA|18.359|0.008|80.089|0.421|0.683|\n\nThis suggests that having access to the correct tokens—regardless of their meaning—is sufficient for the attack. An reconstruction example is shown [here](https://drive.google.com/drive/folders/1LgTor6EGxV5vQWt2TIidvyu6rG5c3-sT?usp=drive_link).\nWe will clarify this distinction in the paper and provide additional examples.\n\nQ7. \"DLG and iDLG..\"\n\nA7. Yes, the results for DLG and iDLG in Table 1 are identical. The only difference is that iDLG has access to the answer (i.e., the ground-truth label). However, as shown in Section 4.2.2, the answer alone is insufficient for reconstruction in DQA, since it lacks the context provided by the question, which guides model attention and gradient flow. As a result, iDLG offers no advantage over DLG in our setting.", "claims_and_evidence": "The main claim is proposing the first gradient inversion attack on multi-modal models, with a novel method specifically tailored for multi-modal DQA models. However, there are concerns in the experiments, undermining the evidence.\n\nThe author claims that the proposed method is the first gradient inversion attack on multi-modal FL setups. However, I found [r1], which also explores gradient inversion in multi-modal FL setups, even though it does not specifically focus on DQA models.\n \n[r1] Liu, Xuan, et al. \"Mutual Gradient Inversion: Unveiling Privacy Risks of Federated Learning on Multi-Modal Signals.\" IEEE Signal Processing Letters (2024).", "methods_and_evaluation_criteria": "The proposed method is ok, but some concerns:\n\n- The paper does not have any theoretical analysis.\n\n- The authors aim to perform a gradient inversion attack in multi-modal federated learning (FL) setups. However, the FL setup used in the experiment is unclear. Neither Donut nor LayoutLMv3 are FL models, and the number of local machines in the FL setup is also unspecified.", "theoretical_claims": "no theoretical claims", "experimental_designs_or_analyses": "Section 4.2.2 (Priors Effect) aims to analyze the contribution of each training loss term. However, the experiments are conducted by adding losses together rather than evaluating each loss term individually. As a result, the findings demonstrate the effectiveness of combining training losses rather than the specific contribution of each loss term.", "relation_to_broader_scientific_literature": "The paper conducted a study of gradient inversion in FL, which is a type of privacy attacks, for multimodal setups, while previous work focuses mainly on unimodal setups.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "see above.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "1. In an FL setup, a gradient inversion attack can be performed at any iteration t. How does t impact the results? For example, does executing the attack at the beginning of training yield lower success rates compared to performing it at the end of training? What value of t was used in the results presented in the paper? Was the attack performed at a specific iteration, or were multiple values of t tested?\n\n2. I am a bit confused by Section 4.2.2 (Question-Answer Effect). The results indicate that document reconstruction is infeasible without question-answer knowledge. However, the authors also state that the specific content of the question and answer is irrelevant to the attack’s success (lines 366-370, right column). How do the authors arrive at this conclusion? Is there any evidence supporting this claim?\n\n3. In Table 1, are the results for DLG and iDLG truly identical? Could you clarify whether they have the exact same values?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "We7TJdnjag", "reviewer": "Reviewer_pbTk", "summary": "The paper explores gradient inversion attacks targeting multi-modal Document Visual Question Answering (DQA) models in the context of federated learning and propose GI-DQA a novel method that reconstructs private document content from gradients. The approach seems to expose privacy vulnerabilities.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his time, effort, and valuable suggestions.\n\nQ1. \"access to a template..\"\n\nA1. While the use of a template may appear simplified at first glance, we argue that this setup is practically grounded and reflects real-world scenarios. In many document-based applications, such as invoices, medical forms, boarding passes, or receipts, the layout and structure are highly standardized across users, with only the sensitive fields (e.g., names, amounts, dates) varying.\nFor example, a receipt issued by a vendor often follows the exact same format for all customers. If an attacker has previously received such a receipt or obtained a sample from another user, it effectively serves as a valid template when attempting to reconstruct another client’s document. This assumption is especially realistic in settings where the attacker is an insider (e.g., the server operator) or has access to public document templates shared across users.\nWe will clarify this practical motivation in the final version of the paper.\n\nQ2. \"QA reconstruction..\"\n\nA2. We thank the reviewer for this important observation. The reconstruction of the question-answer (QA) pair is indeed a critical step in our pipeline, and we address this in Section 3.3.\nIn our current work, we assume access to the QA pair, based on strong evidence from prior analytic-based methods—particularly DAGER (Petrov et al., 2024)—which showed that QA tokens can be accurately recovered from gradients in small-batch federated learning, achieving 100% recovery.\nOur focus is on the document reconstruction step conditioned on known QA pairs, reflecting a realistic threat model where such tokens are obtainable using existing techniques. To strengthen this connection, we will include the QA reconstruction in the final version of the paper and clarify the assumption more explicitly.\n\nQ3. \"How can PIIs..\"\n\nA3. Please see Reviewer fLco, Answer #5—specifically the [visualization](https://drive.google.com/drive/folders/1NxT7gBMSH5RQb9V1bEAuTT2AvXPqhex1?usp=drive_link) of the cross-attention maps, which illustrates how private data outside the question and answer can still influence the computed gradients, offering insight into how such regions are captured.\n\nQ4. \"Until page 5..\"\n\nA4. The sentence in question was intended to convey an intuition about the general difficulty of reconstructing small-sized text regions (such as names or dates), which typically occupy only 1–2% of the image area and thus contribute weakly to the overall gradient signal—especially when compared to larger visual elements like headers, tables, or logos.\nImportantly, our attack is able to reconstruct personally identifiable information (PII) in the document regardless of the specific question or answer, as long as the attack is performed on a model initialized with random weights.\n\nQ5. \"..threat model..\"\n\nA5. \"Our threat model is consistent with most prior gradient inversion works, where the adversary is assumed to have access to the model architecture, parameters, optimizer, and input-output format—but not the sensitive input data itself. The only additional assumption in our setup is access to a template document, which reflects a realistic scenario in many applications (e.g., invoices, receipts, forms) where the document structure is shared across users and only a few fields contain private information. This assumption is grounded in practical document workflows.\"\n\nQ6. \"PIIs are already known..\"\n\nA6. While the answer in the QA pair may reveal a piece of PII, it typically represents only a small portion of the sensitive content (e.g., a single name, date, or value). Many other PII fields remain hidden and can still be reconstructed from gradients.\nOur attack can also reconstruct text outside the question and answer, due to the model’s cross-attention attending to broader regions—especially in early training, when attention is more diffuse.\nAdditionally, see Reviewer fLco, Answer 6, for an experiment showing that PII can still be reconstructed even when the QA tokens are replaced with random tokens.\nWe provide sample data, including documents and their corresponding Q-A pairs, [here](https://drive.google.com/drive/folders/1Xo0u9kZhgOM6AYnoUlUqwdSmoN_psRDq?usp=drive_link).\n\nQ7. \"results without templates..\"\n\nA7. Thank you for the question. We conducted an experiment where the attack was performed without access to a template, and found that reconstruction quality dropped significantly. Without the template, the model must infer both the layout and the content, making optimization far more difficult—especially in structured documents like receipts or forms.\nWe see this work as a step toward full document reconstruction, and future work could explore removing the template assumption entirely.", "claims_and_evidence": "- the results are pretty convincing in terms of numbers and vizualization of the documents\n\nThe threat model however is not very clear:\n\n- Having access to a template (which in the set-up is the document minus the private information) seems toy. \n- The authors do not detail how the question-answer pairs are reconstructed before performing the image reconstruction, while it is apparently the most important step to get good results\n- How can PIIs which are not part of the question/answer be reconstructed in the document? it should not be part of the gradient?", "methods_and_evaluation_criteria": "Having access to the template and perfecty reconstructing the question/answer pairs is crucial to get the results. The first one would not be realistic in practive, and the second one is not explained how.", "theoretical_claims": "no theoretical claim", "experimental_designs_or_analyses": "see above", "relation_to_broader_scientific_literature": "I dont know much about the gradient inversion litterature but it seemed well discussed.", "essential_references_not_discussed": "I dont know much about the gradient inversion litterature", "other_strengths_and_weaknesses": "The paper is well written and easy to follow. \n\nFor the weaknesses, see above Until page 5, it would have been nice to have some numbers to justify some statements.\n\nFor instance : “Visual Text Prior. Accurately reconstructing small-sized words (∼1-2% of the full image) is a highly challenging task due to their minimal contribution to the overall gradient and their low visibility in the reconstructed image” is from the intuition of the authors. My intuition would have been that because the question is specifically about these numbers, then it should be easier\n\n\n\nThe threat model is that the adversary has everything except the sensitive data, not just a general template\n\n\nHow are the q-a pairs reconstructed before the reconstruction of the document? If q-a pairs are already reconstructed, then all the PIIs are already known? Can text be constructed that was not part of the answer? Can you show examples of these qa?", "other_comments_or_suggestions": "- Until page 5, it would have been nice to have some numbers to justify some statements. For instance : “Visual Text Prior. Accurately reconstructing small-sized words (∼1-2% of the full image) is a highly challenging task due to their minimal contribution to the overall gradient and their low visibility in the reconstructed image” is from the intuition of the authors. My intuition would have been that because the question is specifically about these numbers, then it should be easier\n- The threat model is that the adversary has everything except the sensitive data, not just a general template", "questions_for_authors": "- How are the q-a pairs reconstructed before the reconstruction of the document? If q-a pairs are already reconstructed, then all the PIIs are already known? \n- Can text pixels be constructed even if the text was not part of the answer? How is it possible? Can you show examples of these qa?\n- What would be the results without the so called tempates?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k4lS11b5nX", "reviewer": "Reviewer_xopj", "summary": "This paper studies gradient inversion (GI) attacks specifically for multi-modal Document Visual Question Answering (DQA) models and proposes GI-DQA, a novel method for reconstructing private document content from gradients. The empirical experiments demonstrate that their approach exposes critical privacy vulnerabilities even in sota DQA models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his time, effort, and valuable suggestions.\n\nQ1. \"Tab. 2 cumulative..\" + Q3.\"..channel and pixel-level TV..\"\n\nA1. We followed the standard practice of prior works such as GradViT (Hatamizadeh et al., 2022) and GradInversion (Yin et al., 2021), which apply priors incrementally and in combination rather than in isolation. These methods recognize that priors are often **interdependent**—for example, Total Variation enforces smoothness, while Laplacian sharpens edges, and together they produce better reconstructions than either alone.\n\nOur goal in Section 4.2.2 was to show how **combinations of priors** improve reconstruction quality, particularly in **multi-modal DQA** settings, where each prior targets different aspects (e.g., edges, smoothness, spatial layout).\n\nThat said, we agree that isolating each prior provides additional insight. We have therefore conducted an **ablation study** evaluating each prior individually alongside the core gradient loss ($L_{grad}$).\n|Experiment|PSNR|FFT2D|MSE|Binary|Fuzz Ratio|\n|-|-|-|-|-|-|\n|$L_{grad}$|14.619|0.023|81.96|0.198|0.146|\n|$R_{TV-C}$|20.083|0.007|72.446|0.510|0.745|\n|$R_{TV-S}$|18.608|0.008|77.744|0.479|0.749|\n|$R_{TV}$ (combined)|20.527|0.005|72.579|0.579|0.821|\n|$R_{txt}$|17.738|0.011|80.052|0.425|0.680|\n|$R_{gau}$|18.139|0.009|80.907|0.410|0.694|\n\nThe results show that while each prior contributes modestly on its own, the combination (presented in the paper) consistently outperforms any single prior, confirming the complementary nature of these losses. \nFor the TV variants, we can see they contribute meaningfully on their own, but their combination consistently yields the best performance across all metrics.\nNote that these results differ slightly from those in the paper, as they were conducted on a data subset due to rebuttal time constraints. However, they clearly convey the main trend, and full results will be included in the final version.\n\nQ2. \"gaussian essential..\"\n\nA2. We would like to clarify that the absolute values of the loss weights do not reflect relative importance, as each term operates on a different scale. For instance, the Gaussian prior yields smaller values than the text loss, so its weight is lower to ensure balanced gradient contributions. Therefore, a smaller weight for the Gaussian prior does not imply that it is negligible—rather, it ensures balanced gradient contributions during optimization across all terms. This normalization prevents any term from dominating due to scale differences rather than actual relevance. \n\nQ4. \"selection of priors..\"\n\nA4. Our priors were selected for their ability to capture complementary and theoretically grounded aspects of the reconstruction process:\n- Laplacian emphasizes high-frequency components, enhancing edge sharpness and fine details—critical for recovering small text.\n- Gaussian applies a low-pass filter to promote global smoothness, stabilizing early optimization and reducing noisy minima.\n- Total Variation (TV) encourages piecewise smoothness while preserving edges:\n  - Spatial TV regularizes differences between neighboring pixels, reducing noise while preserving layout structure.\n  - Channel-wise TV promotes consistency across color channels, helping suppress chromatic artifacts and preserve clean text appearance.\nTogether, these priors form a multi-scale regularization strategy, guiding optimization from low-level structure to high-level semantics and enabling more stable, accurate gradient inversion in DQA models.\nWe will include in-depth discussion in the appendix illustrating their importance.\n\nQ5. \"auxiliary priors scheduler..\"\n\nA5. Unlike prior works (e.g., GradViT), we found that using only ($L_{grad}$) in early iterations was insufficient for reconstructing meaningful text—likely due to the template reducing gradient signals in sensitive regions. To address this, we use a scheduler that starts with strong prior weights and gradually reduces them, stabilizing early optimization. The table below compares this to GradViT’s approach, which delays prior introduction until later iterations:\n\n|Scheduler|PSNR|FFT2D|MSE|Binary|Fuzz Ratio|\n|-|-|-|-|-|-|\n|Ours|18.390| 0.008|80.383|0.410|0.711|\n|GradViT|8.885| 0.079|83.407|0.118|0.017|\n\nAs shown, GradViT’s scheduler struggles to reconstruct meaningful content, especially in terms of semantic fidelity. We will include this comparison in the appendix and integrate the key findings into the final version of the paper.\n\nQ6. \"set hyperparameters..\"\n\nA6. The prior weights were manually set via grid search to optimize reconstruction performance. As noted in Answer 2, their absolute values don’t reflect relative importance due to differences in loss scales. Both Gaussian and TV priors are especially useful early in optimization, promoting smoothness and structure that complement sharper and semantic losses like Laplacian and text similarity.\n\nQ7. \"Notations inconsistency..\"\n\nA7. Thank you, this will be fixed in the final version.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "The proposed method is intriguing, but certain aspects of the methodology and evaluation setup require further clarification. Please refer to the questions section below.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiment design is reasonable, though additional experiments would be beneficial. In particular, the ablation study—specifically the Prior Effects paragraph in Section 4.2.2—does not fully convince me. Please refer to the questions section below", "relation_to_broader_scientific_literature": "This is the first attempt to conduct gradient inversion attacks on multi-modal models, particularly in the novel and impactful setting of DQA models. It opens new directions for multi-modal safety research.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. It is the first attempt to apply gradient inversion attacks to multi-modal models, particularly DQA models.\n1. The empirical results are promising\n\nWeaknesses:\n1.The writing could be improved, particularly in clarifying the insights and analysis behind the design of the objective function in Eqn. 4.\n2. While the method combines multiple techniques effectively, it lacks deep insights into why certain design choices work well beyond empirical success.", "other_comments_or_suggestions": "1. Section 3.3.1 could be written more clearly. A brief outline before delving into each paragraph would improve readability.\n2. A more detailed explanation of the objective function should be included, particularly regarding:\n- How the weights were chosen.\n- Why Gaussian/TV priors were included (the current descriptions are not convincing enough to me).\n3. Notations inconsistency: (e.g., both $R_{txt}$ and $R_{lap}$ are used inconsistently).", "questions_for_authors": "1. Are the rows in Tab. 2 cumulative comparisons? For example, in the row labeled $+ R_{lap}$, do the numbers compare $L_{grad}$ with $L_{grad} + R_{lap}$, or $L_{grad} + R_{TV}$ with $L_{grad} + R_{TV} + R_{lap}$ ? If the latter, it would be helpful to include a direct comparison for each individual component.\n\n2. Is the Gaussian filter essential? I understand that Gaussian filtering preserves smoothness and prevents overfitting to noise, but\n- the provided examples do not show substantial improvement.\n- the interpretation of Table 2 is unclear due to my first question.\n- the weight $\\alpha_{gau}$ is 0.005, while $\\alpha_{txt}$ is 0.5 and $\\alpha_{TV}$ is 0.05. Given these values, I wonder how important the Gaussian filter actually is.\n\n3. In your overall auxiliary priors, you set weights for each prior. However, for TV, you did not differentiate between channel and pixel-level TV. Was there a reason for this decision? Also, it would be great to include the importance of channel/pixel TV respectively in your Tab. 2.\n\n4. Can you provide more insights into the selection of priors and the scheduler? The current description in Sec 3.3 and Sec. 4.2.2 are not convincing enough to me. A discussion in the appendix illustrating the importance of each prior and your choice of scheduler would be valuable.\n\n5. You mentioned that “auxiliary priors play a critical role in stabilizing the early stage.” Including a plot or table illustrating this observation would be beneficial. Additionally, why does this occur? Most prior work delays activation of auxiliary priors until later stages of optimization to avoid suboptimal convergence. Wouldn’t your scheduler lead to suboptimal convergence?\n\n6. How were the weights $\\alpha$'s chosen? Are they learnable parameters or manually set hyperparameters? Based on my current understanding of Table 2, the example you provided, and the coefficients, it seems that Gaussian/TV are the least important priors. If these weights are manually set, why were Gaussian/TV intentionally downweighted? Are they truly necessary?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a gradient inversion attack specifically designed for multi-modal models used in DQA and making use of known template of the documents.

The reviews are somewhat mixed but mostly positive. The reviewers are generally happy with the novelty and execution of the work and raise no serious criticisms that the authors could not address in the rebuttal.

While the paper might also be a good fit to a more security-oriented venue, I feel that the demonstration of larger than expected vulnerability of federated learning of multimodal DQA models trained with structured data to gradient inversion will be a useful contribution to the machine learning community and recommend acceptance.
Decision: Accept (poster)