NeurIPS 2023 Competition: Privacy Preserving Federated Learning Document VQA
Authors: Marlon Tobaben
Mohamed Ali Souibgui
Rubèn Tito
Khanh Nguyen
Raouf Kerkouche
Kangsoo Jung
Joonas Jälkö
Lei Kang
Andrey Barsky
Vincent Poulain d'Andecy
Aurélie JOSEPH
Aashiq Muhamed
Kevin Kuo
Virginia Smith
Yusuke Yamasaki
Takumi Fukami
Kenta Niwa
Iifan Tyou
Hiro Ishii
Rio Yokota
Ragul N
Rintu Kutum
Josep Llados
Ernest Valveny
Antti Honkela
Mario Fritz
Dimosthenis Karatzas
Venue: Accepted by TMLR ()

Abstract: The Privacy Preserving Federated Learning Document VQA (PFL-DocVQA) competition challenged the community to develop provably private and communication-efficient solutions in a federated setting for a real-life use case: invoice processing. The competition introduced a dataset of real invoice documents, along with associated questions and answers requiring information extraction and reasoning over the document images. Thereby, it brings together researchers and expertise from the document analysis, privacy, and federated learning communities. Participants fine-tuned a pre-trained, state-of-the-art Document Visual Question Answering model provided by the organizers for this new domain, mimicking a typical federated invoice processing setup. The base model is a multi-modal generative language model, and sensitive information could be exposed through either the visual or textual input modality. Participants proposed elegant solutions to reduce communication costs while maintaining a minimum utility threshold in track 1 and to protect all information from each document provider using differential privacy in track 2. The competition served as a new testbed for developing and testing private federated learning methods, simultaneously raising awareness about privacy within the document image analysis and recognition community. Ultimately, the competition analysis provides best practices and recommendations for successfully running privacy-focused federated learning challenges in the future.

Published in Transactions on Machine Learning Research (06/2025)
NeurlPS 2023 Competition:
Privacy Preserving Federated Learning Document VQA
Marlon Tobaben 1x, Mohamed Ali Souibgui', Rubèn Tito', Khanh Nguyen?, Raouf Kerkouche", Kangsoo Jung", Joonas Jälkö', Lei Kang", Andrey Barsky², Vincent Poulain d' d'Andecy', Aurélie JOSEPH
Competition Organizers
University of Helsinki, *Compuler Vision Center, Universital Autònoma de Barcelona,
CISPA Helmholls Center for Information Security, *INIAA, Yooz
Aashiq Muhamed Kevin Kuo Virginia Smith", Yusuke Yamasaki", Takumi Fukami?, Kenta Niwa?, Lifan Tyou', Hiro Ishif', Rio Yokota", Ragul Noa, Rintu Kutum 9a,96 Winning Competition Participants
Carnegie Mellon University, 7 'NTT, Institute of Science Tokyo,
on Department of Computer Science, and Mphasis AI & Applied Tech Lab at Ashoka, Ashoka University, oh Koita Centre for Digital Health Ashoka (KCDH-A), and Trivedi School of Biosciences, Ashoka University
Josep Llados?, Ernest Valveny?, Antti Honkela', Mario Fritz", Dimosthenis Karatzas²
Competition Organizers
University of Helsinki, *Computer Vision Center, Universitat Autônoma de Barcelona, CISPA Helmholtz Center for Information Security
Reviewed on OpenReview: o/perre/eeeeeeveeeee net/ forum? id- 3HKNue
Abstract
The Privacy Preserving Federated Learning Document VQA (PFL-DocVQA) competition challenged the community to develop provably private and communcatiinn-efician solutions in a federated setting for a real-life use case: invoice processing. The competition introduced a dataset of real invoice documents, along with associated questions and answers requiring information extraction and reasoning over the document images. Thereby, brings together rescarchers and expertise from the document analysis, privacy, and federated learning com- munities. Partieipants fine-tuned pre-trained, state-of-the-arl Document. Visual Question Answering model provided by the organizers for this new domain, mimicking typical federated invoice processing setup. The base model is a multi-modal generative language model, and sensitive information could be exposed through either the visual or textual in- put modality. Participants proposed elegant solutions to reduce communicatio costs while maintaining a minimum utility threshold in track and to protect all information from each document provider using differential privacy in track 2. The competition served as a new testbed for developing and testing private federated learning methods, simultaneously raising awareness about privacy within the document image analysis and recognition com- munity. Ultimately, the competition analysis provides best practices and recommendations for successfully running prouuey-focued federated learning challenges in the future.
1 Introduction
Automatic document image processing has become a highly active research field in recent years (Appalaraju
et al. 2024 Lee et al. 2023 Tito et al.) 2023), with invoices being one of the most frequently processed doc- ument types Simsa al. 2023). In a typical real-life invoicing scenario, business suppliers produce invoices
*This analysis jointly written by organizers and participants. author contributions on Page

---

Published in Transactions on Machine Learning Research (06/2025)
for their services and send them to their customers. These documents contain sensitive information, such as consumer caauumerrppuun /purchaser identity, transaction details, purpose, date, phone numbers, amount paid, account information for payment, etc. The customers (document users) need to extract this information and take the corresponding actions (i.c. reject, or make a payment against the invoice). In automated pipelines, these documents would be sent, to AT Lechnology providers, lypically offered in the form of cloud service which automatically extract all required information from the documents, and return it to the document users.
A generic approach to extract information from invoices is DocVQA (Mathew et al., 2020 The extraction is done by asking questions in a natural language form to get specific information as answers, using deep learning model. However, training an accurate DocVQA model requires a considerable amount of data, that is rarely held by single entity. One solution is to train this model collaboratively by aggregating and centralizing data from a set of clients that face the same problem. But, documents often cannot be freely exchanged due to the sensitive information they contain. Federated Learning (FL.) is a learning paradigm that purports to solve this problem (MeMahan et al.) 2017b). Rather than exchanging privatcly-held data, participating entities (known as clients) train models their data in a recentralized fashion, exchanging only the local model updates with a central server. However, even though FL is more private than the centralized approach, a significant amount of information can still be inferred from the updates shared during training, or from the parameters of the resulting trained model, whether by an adversarial server, client, or downstream user (Sikandar et al.
2023
Differential Privacy (DP) (Dwork et al. 2016 is considered the gold standard in terms of privacy preservation and can be used to provide provable privacy guarantees. DP formally quantifies the maximum information leakage from the inclusion of any one individual record in a dataset. Deep learning models can be trained under DP by clipping parameter updates and adding noise to them (Rajkumar & Agarwal 2012 Song et al. 2013 Abadi et al. 2016). However, this introduces a trade-off between privacy and utility. Stronger privacy guarantees require introducing more noise, which proportionately degrades model accuracy.
Another drawback of FL is the high communication cost (Kairouz et al. 2021b). At each federated round, the global model is transmitted by the server to selected clients (downstream step) to be trained on their local data, and then the update of this model is sent by these selected entities back the server (upstream step). For models with millions even billions of parameters, this requires significant bandwidth, multiplied by the number of federated rounds required to reach model convergence.
In this paper, we present an analysis of the NeurIPS 2023 competition on privacy preserving FL Doc that we designed to expose the above challenges and invite the community to design novel creative solutions for this real-life use case. It. brought. together researchers and expertise from the document analysis, privacy, and FL communities. Additionally, added a realistic use case for privacy and FL rescarchers as well as expanding the scope of document analysis to DP solutions.
2 Related Work
Document Visual Question A Answering (DocVQA) Doc has been an evolving field during the last few years. This is due to the emerging datasets that address different document domains. For instance,
industry documents (Mathew et al. 2020 2021 Tito et al., 2021b 2023), infographics (Mathew et al. 2022). multidomain (Landeghem et al. 2023a by, open-ended questions Tanaka et al. 2021), multilingual (Qi el al.
2022), multipage (Tito et al. (023) collections of documents
Tito et al.
202 However, these datasets
are often small and highly domsin-specific, which limits generalizabilie
Federated Learning (FL) FL (Shokri & Shmatikov 2015 McMahan et al.
2017b addresses this issue, applications (Li et al. 2020), particularly in
and has seen practical use in both research and industrial domains where sensitive data is common, such as medicine (Dayan et al.] 2021) and finance (Long et al. 2020). FL carries trade-off between model utility, data privacy, and communication efficiency (Zhang al, 2023 and requires specific consideration of client data heterogeneity, scalability, and fault tolerance. Much recent work in FL focuses mitigating these problems, primarily through developments in aggregation
Automatic document processing services offered by large corporations (AWS Intelligent Document Processing Google Cloud
Document AT Mierosoft Azure Form Recognizer etc) or specialized providers.
2

---

Published in Transactions on Machine Learning Research (06/2025)
algorithms MMohawrab et al. 2023 Elkordy & Avestimehr 2022; So et al.) 2022 Nguyen et al. 2022 but also in parameter compression (Tang et al. 2019) and quantization (Xu et al. 2022).
Privacy Attacks While FL offers privacy advantages, it remains vulnerable to various attacks that jeop- ardize client dataset privacy. In the federated architecture, both the server and clients can potentially act as adversaries. Gradient updates in FL, have the potential to disclose information about the training data, making them susceptible to 'gradient inversion attacks' (Zhu et al. 2019 Zhao et al., 2020 Fu et al. 2022]
Wainakh et al. 2022 Li et al.
2022b Geiping et al., 2020 Melis et al. 2019 Li et al., 2022d), which en-
able accurate data reconstruction. Moreover, adversaries can execute 'membership inference attacks' (Nast
ct al._2........
2019; Mclis ct al.) 2019 Suri ct al. 2022: Shokri et al. 2017 Choquetre-Choo et al. 2021 Li& & Zhang
2021
Hu et al. 2022b) to infer the inclusion of specific data points in other participants' datasets, as well 2019) to deduce subgroup statistics despite secure aggrega- as 'property inference attacks' (Melis et al.
tion (Kerkouche et al.
2023 Pejó & Biczók
2023 y. FL inherently lacks protection against these threats,
necessitating explicit mitigation strategies safeguard client data from adversarics.
Differential Privacy (DP) DP (Dwork et al. 2016 can be used to mitigate privacy attacks. A stochastic algorithm is DP when the output distribution is similar on similar datasets. (s, 6)-DP (Dorrk et al. 2006). which is relaxation of the original s-DP and introduces the additional paramcter 6, has a privacy budget that bounds how much the output distribution can differ between similar datasets. The privacy budget. consists of E 2 0 and 8 E [0, 1], where smaller values correspond to a stronger privacy guarantee. DP guarantees also depend on the definition of similar datasets, called adjacent datasets. These adjacent datasets can differ on different levels, which can be for example the inclusion or exclusion of one datapoint (e.g., an image or document) the inclusion exclusion of a client (e-g, hospital). Especially relevant to setting is group-level DP, which preserves privacy leakage from the inclusion exclusion of groups of datapoints (Galli ct al.
2023 Marathe & Kanani 2022 such as multiple records associated with a specific user.
Training ML models under DP Training models under DP is often done with some form of DP stochastic
gradient descent (DP-SGD) (Rajkumar & Agarwal 2012 Song et al. 2013 Abadi et al., 2016) that has some modifications to non-DP SGD. The core differences are al each step: (i) set of datapoints is sampled from the training dataset, (ii) per-example gradients are computed and clipped so that their L2-norm does not to not exceed certain clipping threshold, (iii) the per-example gradients are accumulated, (iv) and Gaussian noise is added to the accunulated gradients. The practice DP-Adam other optimizers can be used that have the same modifications. For language models specific optimization methods like DP-Forward (Du et al. 2023) can be beneficial. In FL, methods like DP-FedAvg (McMahan et al.) 2018) can be used and DP can be combined with Secure Aggregation to improve the utility (Truex et al. 2019p. Furthermore, DP-FTRL (Kairouz et al., (202Ta) adds correlated noise which can in some cases improve the utility /privacy trade-off. We refer to Ponoomarev al. (2023) for a comprehensive introduction to DP in Machine Learning.
High utility models under DP DP introduces trade-off between privacy and utility that can make it hard to use DP in some applications (Ponomareva et al.] 2023). Currently, many works improve the utility- privacy tradc-off through transfer learning (Yosinski et al. 2014) assuming the availability of non-sensitive public data for pre-training and only utilixing DP to protect sensitive rewn data during fine-tuning. We would like to refer to Tramer et al. (2024) for a discussion on the drawbacks these assumptions. Because of these advancements and the popularity of the approach this competition also assumes the availability of a model checkpoint that has been pre-trained on public data. This enables, transfer learning, which is highly
effective for both language [Li et al. 2022c Yu et al. 2022 and vision tasks (Cattan et al., 2022 De et al. 2022 Kurakin et al. 2022 Tobaben et al. 2023 In particular, parameter-efficient fine-tuning (Houlsby
ct al. 2019) with adaptation methods such as LoRA (Hu et al.] (022a) have been demonstrated to yield improved utility-privacy trade-offs for DP, as have quantization (Youn et al. 2023) or compression of model updates (Kerkouche et al. 2021a by Miao et al.. 2022). All these methods reduce the size of the updates, and thereby reduce the amount of noise addition required. The same strategies often yield competitive performance for FL.
3

---

Published in Transactions on Machine Learning Research (06/2025)
3 General Competition Information
This section describes general information about the competition that is common to both tracks. These are the dataset, metrics, model, starter kit and the participation statistics.
3.1 PFL-DocVQA Dataset
For this competition, used the PFL-DocVQA dataset (Tito et al. 2024), the first dataset for private federated DQA The dataset is created using invoice document images gathered from the DocILE dataset (Simsa et al. 2023). For every image, it contains the OCR transcription and a set of question/answw pairs. The version used in this competition contains a total of 336,842 question-answer pairs framed on 117,661 pages of 37,669 documents from 6,574 different invoice providers (See statistics in Table A1 and split of documents per provider and client in Figure A.2). The original PFL-DocVQAA datasct designed to be used in two tasks, and so is divided into two subsets. The first task aims training and evaluating machine learning privacy-preserving solutions on DocVQA in FT. fashion and uses a base subset of PFL--DoCQQA called the "BLUE". The second task aims at designing membership inference attacks to assess the privacy guarantees of the DocVQA models that were trained with the data of task one. These attacking approaches are to utilize a second subset called the "RED" data. In this competition, we focus on the first task, thus, we use only the "BLUE" data. For more details on the full PFL-DocVQA datasets, refer to Tito et al. 2024
The competition aims to train and evaluate DocVQA systems that protect sensitive document information. In our scenario, sensitive information encompasses all information originating from each invoice provider. Therefore, effective model must prevent the disclosure of any details associated with these providers (such provider names, emails, addresses, logos, etc.) across diverse federated clients. Following this, the base data used in this competition consists of a training set divided among N clients (we use N = 10), validation set and a test set. (See Figure A. The training set of each of the N clients contains invoices sampled from a different subset of providers, resulting in a highly non-i.i.d. distribution. In the validation and test sets, docunents both from the providers that were seen during training, and from a set of providers that were not seen are included, to better evaluate the generalizability of the models.
3.2 Evaluation Metrics
In the PFL-DocVQA Competition three main aspects are evaluated: The model's utility, the communication cost during training and the DP privacy budget spent through training the model.
Utility To evaluate the visual question answering performance of the participants' methods we use accuracy and ANLS (Average Normalized Levenshtein Similarity). standard soft version of accuracy extensively used
in most of the text-based VQA tasks (Biten et al. 201 9a b Mathew et al. 2020 Tilo et. al.
2021b Mathew
et al.
2021 Tito et al. 2021a Mathew et al. 2022 Tito et al. 2023 Landeghem et al.)
2023b a). This
metric is based on the normalized Levenshtein Distance (Levenshtein) 1966) between the predicted answer
and the ground truth, allowing us to assess the method's reasoning capabilities while smoothly penalizing
OCR errors.
Communication cost We measure the efficiency of the communications the total amount of information transmitted between the server and the clients in Gigabytes (GB) in both directions. The initial transmission of the pre-trained model to the clients is not included in the communication cost.
Privacy The methods of track are required to comply with a DP privacy budget of no more than a pre-defined e E {1,4,8) at 8 = 10 10-5, We provided a script within the starter kit detailed in Section 3.4 to compute the required noise multiplier given the target (c, 8). Participants may need adjust the script to their algorithms. Moreover, we required the participants to upload a theoretical privacy proof of their methods, which manually reviewed by the competition organizers.
4

---

Published in Transactions on Machine Learning Research (06/2025)
3.3 Pre-trained Model
The participants were asked to implement their solutions starting from the same pre-trained model. The architecture chosen is Visual T5 (VT5), it is a multimodal generative network consisting of simplified version of Hi-VT5 (Tito et al. 2023), which was originally proposed for multi-page DocVQA. VT5 exploits the image and text modalities, which is beneficial to perform the DocVQA task. However, this dual-modality approach also presents more complex challenge: safeguarding private information across both modalities, compared to handling just one. Moreover, VT5 is generative model based on the T5 (Raffel al. (0200) language model. Language models can suffer hallucinations (Rawte et al. 2023). leading to the potential leakage of private information.
The architecture VT5 consists of an encoder-decoder model based on T5. The input of the model is the question, the OCR tokens of the document (text and spatial information), and the encoded document image using the DiT (i et al., 2022a) vision transformer model. These three inputs are concatenated and fed to the VT5 to output the answer following the autoregressive mechanism.
We also provide pre-trained weights for VT5. First, the language backbone T5 is initialized with the pre- trained weights on the C4 dataset (Raffel et al. 2020), and the visual DIT with the pre-trained weights the document classification task. After that, the full model is finc-tuned on the single-page DocVQA task, using the SP-DocVQA dalaset (Mathew et al.
2020 2021 for 10 epochs.
3.4 Starter Kit
The starler kit includes the pre-trained model checkpoint, the fine-tuning datasel, code for running the baselines and instructions on how to run and modify the code. The code itself is based on es- tablished librarics such as PyTorch (Paszko et al. 2019) and the FL framework Flower (Boutel et al. 2020). Besides the training code, the starter kit includes functions for computing the privacy parame- ters based on the hyperparameters and for logging the communication between server and clients. We tested the installation and execution of the baseline on various clusters across different institutions and provided support to participants if they encountered any difficulties. The starter kit is openly available: https: htwpthhwwwo//uuueeee/////-------
3.5 Participation Statistics
Refer to Table for the participation statistics. Our competition has gained interest across the munities and remains open benchmark in the future: htttps:/benncmarkss www.echmmrrs.eesaaaii....o silaa-aaieu//cch226oom introduction In Section 6.2 we discuss measures to lower the participation threshold.
Table 1: Participation Statistics as of May 31, 2024.
Registrations platform Downloads
Countries Submissions Track Submissions Track
382
494
21
13
4
Track 1: Communication Efficient Federated Learning
Track focuses on training high utility models while reducing the communication cost in federated learning. We describe the task, the organizers' baseline and two submitted approaches (See Table 2
4.1 Task Formulation
The objective of track 1 is to reduce the communication used (# bytes), while achieving a comparable utility (ANLS) with the organizers' bascline. The bascline achieved a validation ANLS of 0.8676 and we define a comparable utility to the baseline as 0.8242 ANLS (5% w.r.t. the baseline). Any submission that achieves least that ANLS is valid, thus the deciding factor for winning the competition is the communication efficiency,
5

---

Published in Transactions on Machine Learning Research (06/2025)
which is measured using single metric. We opted for scoring using single metric as the trade-off between utility and communication is not linear. Furthermore, in real world applications less communication efficiency will lead to higher monetary costs or longer training times that need to be considered in contrast to changes in model utility.
Participants are required to use the VT5 baseline model with the initial pre-trained weights and utilize only the PFL-DocVQA dataset for fine-tuning. Further the participants are not allowed to change the PFL- DocVQA data distribution. Additionally, participants between the clients and the central party (# bytes) and the final model checkpoint.
required to upload log of the communication
The organizers evaluate the model utility on a secret test set and thus the model architecture needs to be the same as the initial baseline. While this makes some solutions such model distillation more challenging, the track is open to a wide range of possible solutions. Participants could, e.g., utilize parameter-efficien fino-tuning, compression of the FL updates, lower precision or better hyper-parameters to achieve higher communication efficiency while maintaining a comparable utility.
4.2 Baseline Solution Track
The baseline solution for track 1 fine-tunes all parameters the pre- pre-trained model but the visual module. It essentially uses Federated Averaging (FedAvg) (McMahan et al. 2017a). In each global round, the central server samples K 2 clients out of all N 10, and cach of these clients computes the weight updates locally across multiple local rounds. The central server aggregates the client updates and communicates the updated model to the sampled clients in the next round. This baseline achieves 0.8676 of ANLS and 77.41 accuracy the validation set after 10 FL Rounds. It transmits 1.12GB constantly for each communication stream, which results in total of 44.66GB during the entire training process. We sample K = 2 clients at every federated round.
Table 2: Competition Winners Track 1 (Communication efficient federated learning)
Rank
Team
Method Communication 4
ANLS t
1
Muhamed al. (Soction 4.3
LoRA
0.38 GB (-99.14%)
0.8566 (-3.45%)
2 Niwa et al. (Section 4.4
FedShampoo
10.01 GB (-77.37%)
0.8891 (+0.20%)
Organizers (Section 4.2
Baseline
44.65 GB
0.8873
4.3 Winner Track 1: Muhamed, Kuo, and Smith
We considered three orthogonal methods to reduce communication (LoRA, tuning FL hyperparameters, and quantization). The winning solution for Track uses only LoRA (100x reduction). Combining all methods can achieve a 5200x reduction. We present an overview here and discuss the details in subsections.
Full finetuning
40.05 GB
LoRA
0.38 GB
399.67 GB
(-99.04%)
Tuned HPs
55 MB
0.33 GB
(-99.86%)
Quantization
47 MB
(-99.98%)
1 MB
MB
MB
MB
16 MB
32 MB
64 MB
128 MB
256 MB
512 MB
1.02
2.05
GB
GB
4.1
GB
16.4 GB
32.8
65.5 GB
Communication (log scale)
LoRA. Low-Rank Adaptation trains low-rank adapters while freezing the rest of the model (Hu et al., 2022a).. We use LoRA to reduce the number of trainable parameters to 3.4M (1.37% from 250M). Using 2 clients per round, reach the target ANI.S in 7 rounds (0.38 GB total communication).

---

Published in Transactions on Machine Learning Research (06/2025)
2. Tuning FL hyperparameters. On top of 1. LoRA, sample 1 client per round (default: 2) and train for 16 local epochs (default: 1), which respectively reduces communication and improves utility. With these adjustments, we reach the target ANLS in 2 rounds (55 MB total communication).
3. Quantization a lossy compression approach which we use to reduce the size of the communicated LoRA updates. We use NF4 (4-bit) quantization which reduces the message size by ~ 8x while achieving the target ANLS with the same configuration 2. (7.7 MB total communication).
In all experiments, clients perform local fine-tuning with batch size 16 and learning rate 2e-4. In our code, we train one model at a time using data parallelism. Specifically, we split each batch over 8 GPUs, resulting in a batch size of 2 per GPU (we used 8 GelForce GTX 1080 Ti GPUs). Our code is shared on Github: htllps:/ggihubb.co www.o//iiiiiiiiiiiioo/ooo
4.3.1 Communication cost
Since all messages have an identical size in this FL setting, the total communication cost is simply a product of the a) size of communicated messagos and b) number of messages sent. In the table below, we break down each method's cost using the following equations:
'Total' = 'Message Size'
x Messages'
where 'Message Size' ('LoRA' (#params)
+ 'aas (#params)))
'Bits' (per param)
and "Messages' CC" (clients per round)
x 'R' (FL rounds)
2 (up and down)
Message Size
Messages
Total
ANLS
Method
LoRA
Base
Bits
Bytes
C
Bytes
Val
Test
Baseline
250M
32
1.11 GB
2
10
40 GB
.8676
.8873
LoRA (rank=6)
660K
2.75M
32
13.7 MB
2
7
380 MB
.8400
.8566
Tuned HPs
660K
2.75M
32
13.7 MB
55 MB
.8467
.8683
Quantization
660K
2.75M
4.5
1.92 MB
1
7.7 MB
.8444
.8673
Table 3: We summarize the three methods used. LoRA reduces the number of trainable parameters, tuning HPs reduces the number of messages, and quantization reduces the parameter bitwidth.
LoRA. While the VT5 architecture contains both a language backbone (T5) and vision backbone (DiT), only use LoRA the language backbone and insert 110K new parameters per LoRA rank. For the vision backbone ("Base", we directly fine-tune the spatial encoder (2.16M params) and visual embedding projection head (0.59M params). All other parameters in the entire model are frozen. Although LoRA changes the model architecture during training, can be merged with the pretrained architecture after training is complete, which allowed us to make valid submissions.
The 110K parameters (0.44 MB) per LoRA rank come from applying LoRA to the query and value projections in each attention layer of the language backbone. Each projection matrix has dimension 768 x 768, so its two adapter matrices A, B will both have dimension x There are 36 attention layers which contain query and value projection, giving the final value:
36 (layers) x 2 (query and value) x 2 (A and B) x 768 x r (rank) = 110, 592 ~ 110K x
We note that LoRA typically takes more iterations to train than full fine-tuning, While the full fine-tuning baseline provided by the organizers achieves .8242 validation ANLS in 4 rounds (this is 5% below the .8676 ANLS at rounds), we find that LoRA takes 7 rounds (t 2x) to achieve the same ANLS. However, the parameter reduction from LoRA (L 100x) greatly offsets this cost. For all experiments in this section, we use LoRA with rank 6.
7

---

Published in Transactions on Machine Learning Research (06/2025)
4.3.2 Tuned FL Hyperparameters
We find that extended local fine-tuning additional communication cost. In Table ! we show that training only on single client can achieve .8242 ANLS. We also find that sampling a single client is more efficient than averaging multiple clients each round. In Table 5 Client' usually outperforms '2 Clients' when given double the number rounds.
single client is very helpful, it increases utility with
FL Rounds
Client ID
1 Client
1
2
4
8
Epochs
1
2
1 Epoch
.7419
.7875
.8083
.8331
.7648
.7638
.7577
.7552
2 Epochs
.7719
.8061
.8206
.8382
2
.7893
.7912
.7904
.7797
2 Clients
(2x
communication cost)
.8111
.8108
.8039
.8089
.8247
.8219
.8231
.8176
1 Epoch
.7493
.7899
.8232
.8400
16
.8337
.8345
.8329
.8307
2 Epochs
.7696
.8083
.8355
.8513
Table 4: Extended local training single client greatly improves validation ANLS.
Table 5: Sampling one client and training for double the rounds achieves a higher validation ANLS than sampling two clients.
One surprising takeaway from our experiments is that the data from a single client is adequate to train a competitive model. However, there are many limitations with limiting the client subsample, which briefly outline. First, in cross-device FL scttings which consider large network (up to millions) of clients, extreme subsampling can lead to low-quality global updates. Next, since subsampling slows down convergence, the model will take more rounds and thus more wall-clock time to train. Finally, in the context of privacy, sampling fewer clients makes more difficult to bound the sensitivity of the aggregate update with respect to any individual client's s data, which results in greater privacy loss.
4.3.3 Quantization
By default, each parameter communcattdd as a 32-bit floating-point value (FP32). We reduce this to 4.5 bits (1 7x) by using NF4 (normal-float) quantization (Dettmers el al.] 2023). While NF4 proposes using LoRA on top of a quantized backbone, we use quantization reduce the size of all communicated parameters (in both LoRA and the backbone). Similar recent FL methods have generally explored combining LoRA
with parameter compression to reduce communication Yadav al. 2023 Kuo et al. 2024
In NF4, each parameter is stored using 4 bits (16 unique values) and each block of h: 64 parameters shares an FP32 normalization factor. This adds up to 4 - (32/k) 4.5 bits per parameter, as shown in Table 3 Parameters are quantized only before communication, while finetuning and aggregation are all done in full precision. As we show in Table D quantization slightly harms model performance, but this cost greatly offset by the reduction in communication.
4.4 Runners-up Track: Niwa, Ishii, Yamasaki, Fukami, Tyou, and Yokota
We briefly present our methods and experimental results. For more detailed information can be found in Section 4.4.1 We aimed to achieve faster convergence of training for local models with fewer communication rounds. To achieve this, we utilized Shampoo (Gupta et al.) 2018). second-order optimization method, in local update rules by multiplying the local preconditioning matrix to the local stochastic gradient. The update rules of our method, named FedShampoo, are outlined in Algorithm [ in Appendix C Shampoo enables smooth local updates by geometrically rotating and scaling stochastic gradients. To reduce the memory footprint in computing large-scale preconditioning matrices, we approximated them by employing layer-wise dlocl-diagonalizzatonn Notably, the local preconditioning matrices (approximated by sub-matrices) were not transmitted to the central server, thus avoiding excess communication costs. Furthermore, we
8

---

Published in Transactions on Machine Learning Research (06/2025)
Full-precision
Quantized
Round
Stage
Client
Clients
1 Client
2 Clients
1
Download
Initialize weights using shared RNG seed
Finetuning
.8337
.8341
.8337
.8411
Upload
.8301
.8313
Aggregation
.8555
.8233
2
Download
.2533
Finetuning
.8467
.8437
.8448
8445
Upload
.8444
.8224
Aggregation
.8520
.8518
Total Communication
55 MB
110 MB
7.7 MB
15.4 MB
Table 6: We track the validation ANLS after each stage of communication-effiieen FL. When sampling 2 Clients' per round, 'Finetuning" and 'Upload' refer to the average ANLS over the two client models. '.' indicates that the same mo)cl(s) are evaluated as the coll above e.g. full-precision 'Upload' and 'Download' do not change the model(s).
excluded the embedding layer from the optimization target. resulting a reduction of approximately 26 % in communication per round compared to whole parameter
FedShampoo achieved the target ANLS score with 10.01 GB communication cost. Refer to Figure A.4 in Appendix C for convergence curves using validation loss, ACC and ANLS. We submitted the model after only R = communication rounds, surpassing the larget ANLS score of 0.8873 and resulting in an approximately 30 % reduction of the communication cost compared with the baseline method (using solely AdamW-based optimizer). Furthermore, FedShampoo achieved higher ACC and ANLS scores compared with the baseline method after exceeding the ANLS target score (after communication rounds). This provides empirical evidence of FedShampoo's 's faster convergence, which benefits from applying the preconditioning matrix to the stochastic gradient. The detailed experimental configurations, such as hyperparameter tunings of learning rate and clipping threshold, are summarized in Section 4.4.1
In Table 2
4.4
4.4.1 Details on FedShampoo
Update rules of FedShampoo: First, explain the update rule using Shampoo Gupla al. (2018). As discussed in Section 444. Shampoo is second-order optimization method that involves multiplying the preconditioning matrix with the (stochastic) gradient, and the preconditioning technique in Shampoo is introduced in the local model update in our FedShampoo, which is summarized in Algorithm !1
In the optimization of models in the form of neural networks, it is typical for model parameters to described by a stack of matrices matrices/tensors tensors to transform each layer's input and output. Although we have focused on formulating the update rules in matrix manner (since we will mainly focus on Transformer-basee model), it is not loss of generality. For all clients i E [N] and each layer b E [B], let wib E Rd be the model parameter in the b-th layer of the neural network, and G E ggdoat.adaa.. be the stochastic gradient of the local loss function with respect to (() The local model update rule using Shampoo is given by
(++1) L.b + Gi
(()))
Gi
((+ R!9 + [a()]]
Gi,b
w(++1) w!O [
1/4
1.6
(t)
Rib
1/4
(1)
We submitted model applying LoRA FedShampoo; however, did exceed the target ANLS score.

---

Published in Transactions on Machine Learning Research (06/2025)
where 7/ denotes the learning rate, and 4,)) Rd a X and R. RRdmaxdns are the preconditioning matrices for the gradient and the weight matrix, respectively.
In Equation | the local preconditioning matrices, Libb and Ra,, are multiplied to both sides of the stochastic gradient in a matrix form Gi,b. This process can be interpreted as miligating changes in the local gradient of loss function through model parameter updates by multiplying local preconditioning matrices. This supports mitigating the negative effects of complex loss landscape the loss function using ncural networks, and it can lead to fast convergence to the stationary point.
Thanks to the Shampoo application in laser-wise manner, it is possible to track Lg,b and R1.6 for each layer, which significantly reduces the memory footprint. Specifically, while the full-matrix version of AdaGrad Duchi et al. (2010) requires memory linearly proportional to the number of model parameters O( & Shampoo only roquires memory with O dl for cach layer. Furthermore, the inversion of the preconditioning matrices can be efficient, since takes Od + w rather than O( d m.,) of computational complexity.
terms
Additionally, element-wist clipping was used in the local model update rule, which is a de-facto standard for stable optimization of the Transformer-based models, mentioned in e.g., Zhang et al. (2020). Due to the heavy-tailed noise in stochastic gradient, the magnitude of updates in model parameters has significantly changed, leading to unstable convergence. To address this issue, we effectively alleviated this phenomenon by incorporating the clipping of the magnitude of each element of gradients into adaptive updates using Shampoo.
Finally, as noted in Section 4.4 to reduce the amount of communication per round, the embedding layer was excluded from the optimization target. This results in reduction of around 26 % amount of parameters, rather than transmitting whole paramcters.
In the following, experimental setups are explained.
Compared methods: In our experiment, we utilized two methods with differing local update rules: 1) the bascline method using AdamW optimizer, and 2) FedShampoo using Shampoo- based preconditioner to the Stochastic Gradient Descent (SGD).
To ensure fair comparison of the two methods, several hyperparameters (learning rate 1) and element-wise clipping threshold C`' were empirically tuned. This was done while main- taining fixed values for the total communication rounds R = 10, the number of inner loops for local update L = 5000, and the number of client sampling K = 2. In Figure A.3 a summary of our hyperparameter tuning for FedShampoo is provided. After performing empirical trials, we selected 1) = 2e and C = 0.2.
Hyperparameter Tuning:
Computing enviromment: We used server with 8 GPUs (NVIDIA A6000 for NVLink 40GiB HBM2) and 2 CPUs (Xeon).
Experiment results: The best validation accuracy and ANLS were achieved with the proposed FedSham- poo (with freezing embedding layer). As depicted with two lines, there was confirmed difference between the two mothods.
4.5 Takeaways from Track 1
In the below box we highlight important takeaways from the Track 1. The insights are linked to prior literature that suggested decreasing the communicated parameters (Hu et al. 2022a Tobaben et al. 2023 and the quantization of the communicated parameter update (Yaday et al. 20233 Kuo et al. (024).. In terms of hyperparameters it noteworthy that the the winning team used significantly more local epochs before communicating than the baseline. While the literature (Karimireddy et al.] 2020) suggests that a large number of local updates leads to the clients drifting apart but the findings suggests that 16 local epochs do not lead to this effect and are more optimal than just one epoch as in the baseline.
10

---

Published in Transactions on Machine Learning Research (06/2025)
The participants improved the communication efficiency through multiple orthogonal approaches that could be used complimentary.
Takeaways Track 1:
1. Optimizing the hyperparameters is crucial as highlighted by both winner and runners-up.
2. Carefully selecting the parameters to communicate (e.g,, LoRA) can yield benefits but also significantly decrease performance.
3. Quantization of the communicated parameter update, but not the prior computation and aggregation only slightly harms performance.
4. Changing the optimization method can lead to faster convergence, allowing one to reduce the number of rounds of communication while yiclding similar utility.
It should be noted that these complex methods require substantial compute for ablation studies, as illustrated by the number of GPUs used.
5 Track 2: Differentially Private Federated Learning
Track 2 focuses on training as high utility models as possible while preserving all information from each document provider in the training set through DP. We describe the task, the organizer's baseline and two submitted approaches (See Table 7).
5.1 Track 2 Task Formulation
The objective of track 2 is to achieve the best utility possible while protecting all information from each document provider in the training set, which could be exposed through textual (provider company name) or visual (logo, presentation) information. Prior work (Tito et al.) 2024 Pinto et al. 2024) has shown that non-DP Voc models leaks sensitive information about the trianing dataset. Participants are required to train under DP at different levels from strong formal DP (e 1) to reasonable privacy guarantees (e = 8) to mitigate the risk of provider information being leaked Ultimately, the goal is achieve the best utility while complying to the privacy budgets of [1,4,8) at 5 = 10-5 The definition of DP critically depends the concept of adjacency of datasets. We seek to protect the privacy of providers and thus the typical docunent-level adjacency definition would be too weak, as there are many documents from the same provider and combining them could leak private information. Instead we use provider-level add/remove adjacency, where adjacent training datasets can be obtained by adding or removing all documents from one provider. Prior work denotes this as group-level DP (Marathe Kanani 2022 Galli et al. 2023).
Participants are required to follow the same rules regarding the pre-trained model and fine-tuning data as in track 1. Besides uploading the final model checkpoint solutions, they are required to submit thcoretical privacy proof and description. The requirement for theoretical privacy proof in track 2 ensures that the solutions proposed by participants are rigorously validated for their adherence to differential privacy principles. This proof demonstrates that the final model maintains the privacy of all information from each document provider by offering a quantifiable measure of privacy loss. Additionally, a thorough description and code submission are necessary to facilitate reproducibility and allow for independent verification of the privacy claims, ensuring transparcncy and trustworthiness in the solutions provided.
Verification that the submissions are DP The organizers reviewed all submissions to track 2 to ensure that they are DP. Multiple organizers separately reviewed the description of the method, the privacy proof and the source code. Most of the participants' privacy proofs followed from the baseline proof and thus the review process of the proofs was lightweight. The organizers asked questions about unclear details of the privacy proofs. Otherwise the review process of the proofs was comparable to reviewing ML conference submissions. The source code of the participants was inspected using static code analysis mostly focusing on the deviations from the baseline implementation and DP-SGD critical parts of the implementation, e.g.,
We refer to Section 5.2 in Ponomareva et
2023 for discussion of classifying privacy budgets into tiers.
11

---

Published in Transactions on Machine Learning Research (06/2025)
the clipping operation, the noise addition and the DP accounting. In more complicated settings it would make sense to run unit tests against these parts of the implementation, run the experiments or re-implement but in this challenge the organizers did not need to do that as the amount of confidence in the participants' implementation was sufficient based on the static code review.
5.2 Baseline Solution Track 2
The baseline solution for track utilizes DP-SGD with provider-level add/remove adjacency. The optimiza- tion of the model is done in multiple global rounds. In each round, the central server first samples a set of clients from all N 10 clients. Each selected client runs a local instance of federated learning where each provider acts as the training data of a virtual client within the real client. The client randomly selects providers, clips the per-provider updates and the adds an appropriate amount of noise so that the update aggregated by the server is differentially private with respect to all providers over all clients The privacy loss of the bascline follows the usual analysis of DP-SGD consisting of compositions of sub-sampled Gaussian mechanisms. The loss depends on the number of iterations Tet, sub-sampling rate & (both over clients and providers) and noise scale a MMroonov et al. 2019 Balle et al. 2020). The baseline is obtained through 5 FL, Rounds. It transmits 1.12GB constantly for each communication stream, which results in a total of 22.32GB during the entire training process. We sample K 2 clients per round and M = 50 providers on each client. The updates are clipped to norm of 0.5 and the Gaussian noise is computed that the privacy budgets of E E {1, 4, 8) at 8 = 10-5 is spent at the end of training.
The baseline is DP as stated in Theorem 5.1 For details see the privacy analysis in Appendix
B
Theorem 5.1 (Privacy of FL-GROUP-DP), For any 0 < 8 < and a 2 1, FL-GROUP-DE is (min, (Ta E((llg) + log((a - 1)/a) (log 5 + log - 1)). 6), where Ex((a)g) is defined in Egg 15 9
C-IMI
ming [Gal'
The proof follows from the RDP property of the subsampled Gaussian mechanism, RDP composition and conversion from RDP to approximate DP (Theorems B.6 B.7 B.8) and the fact that group (provider) is sampled in every federated round if (1) the corresponding client is sampled, which has a probability of C, and (2) the batch of groups sampled locally at this client contains the group, which has a probability of at C-MM
most ming [Ce
[M]]
Therefore, a group is sampled with a probability of q ming [G&]
Table 7: Competition Winners Track 2 (Differential Private Federated Learning)
ANLS at 4
Rank
Team
Method
at
at E 8
Ragul N and Kutum (Section 5.3
LoRA
0.5854
0.6121
0.6225
Fukami et al. (Section 5.4
DP-CLGECL
0.5724
0.6018
0.6033
Organizers (Section
Bascline
0.4832
0.5024
0.5132
5.3 Winner Track 2: Ragul N and Kutum
Similar to the winning solution for track 1, our method also uses LoRA. We choose LoRA for the following two reasons: First, it significantly reduces the communication cost as shown in Section 4.3 Second, empir- ical results have shown that differentially private adaptation of language models using parameter-efficient methods such as LoRA outperforms full fine-tuning in centralized settings (Yu et al. 2022). These meth- ods reduce the overall noise added by only updating a small proportion of the parameters in the model, thereby increasing the utility of the model. The communicatio efficiency of LoRA also allowed us to increase the nunber of FL rounds from 5 in the baseline method to 30 in our method without increas- ing communication costs. With these changes to the baseline, our method improved the ANLS by 10-11 percentage points across all privacy settings. The GitHub repository of our solution can be accessed at httgggitta...oooouuuuaaaaaaa KutumLab hat.uuummaaappppoooaaaaaa ptt.ddooooraiiitt---L....
attps://githubb.e com
Note when no clients sampled in FL. round the server still needs add noise.
12

---

Published in Transactions on Machine Learning Research (06/2025)
5.4 uunners-up Track 2: Fukami, Yamasaki, Niwa, and Tyou
We briefly present our methods and experimental results. More detailed information can be found in Sec- tion 5.4.1 It is well-known that applying DP to FodAVG with a relatively high privacy level often stagnates the model training process due to local parameter drift. This is mainly caused by i) noise addition in DP and ti) data heterogeneity among clients. To address these issues, we propose DP-CLGECL, which incorpo- rates the DP's Gaussian mechanism into CLGECL Tyou et al. (2024). The update rules in DP-CLGECL are derived by solving a linearly constrained loss-sum minimization problem, resulting in robustness against local gradient drift. due to data heterogeneity, and this would also be effective in addressing the drift issue due to DP's Gaussian mechanism. Note that the DP analysis of the private baseline detailed in Appendix B is applicable to our DP.CLGELL. More details about our mothodologies are provided in Section 5.4.1
As indicated in Table 7 ANTS showed significant improvement with the use of our DP-CLGECL compared with the baseline method for each E. Associated experimental results, including convergence curves in Figure A.5 are summarized in Section 5.4.1 and Appendix [ After passing the competition deadline, we observed a negative impact of using AdamW optimizer in the baseline method. The norm of stochastic gradient, preconditioned by AdamW, often increased, and the gradient clipping used to ensure the pre- defined DP levels led to a loss of valuable information in model parameter training. To address this issue, we replaced AdamW with momentum in the local update of DP-CLGECL, resulting in further improved ANL.S. Although more details can be found in Figure A.6 the ANLS was then 0.5918 for E using DP-CLGECL with momentum.
5.4.1 Details on DP-CLGECL
Firstly, we provide a brief explanation of the formulation of CLGECL Tyou ct al. (022 For FL consisting of n local clients and a central server, we aim to solve a loss-sum minimization problem with linear constraints local parameters {wo)
n
min
(ww') s.t. w = =w w (Vi E E E E'),
where fr represents the local loss function and {1...., n) E E, i-..t., .n....n] details can be found in Tyou et al. (2024). A solver for equation 2 over the centralized network is referred to as CLGECL, Due to the constraint of identical local parameters, CLGECL is expected to be robust to gradient drift. For this competition, we propose DP-CLGECI, which introduced AdamW as a local update, client sampling, and Gaussian mechanism in DP for CLGECL, as summarized in Algorithm
E The derivation
To follow the regulation of this competition task, specified this operation as follows: that each clicnt's 's data set D₁ is partitioned into a set G2 of disjoint and pre-defined groups, and cach client has different groups. The server randomly selects a subset K of clients in each round to update the global model. Each client receives the global model from the server for each round. The client selects a random subset M of groups, calculates the gradient Auff by SGD with momentun for each group, and the gradient Awf² updated with the dual variables X, clipping into clipped the gradient A An have a bounded T2 norm of S, where S denotes the sensitivity of the gradient Auς. The sum of And for all groups is caleulated and perturbed by the Gaussian mechanism. Finally, the i clients selected by the central server calculate the model update difference w' - W:-1, send it to the server, and update the dual variable A.
First, we assume
Privacy analysis: In the privacy analysis of DP-CLGECL, we aim to determine E and a that ensure that Awf + +W(0, (') guarantees (a, RDDP.. We then apply the composition on the RDP, and convert the RDP to DP. The privacy analysis of PL-GROUP-DP (Mrrathe & Kanani 2022 Galli et al. 2023) demonstrates a a method to guarantee (a, for A oo, This analysis can be applied to our FL-GROUP-DP.
DP-CLGECL guarantec (e, 8DP if o is used, satisfying the following
= min Ex(al q) + log((a- - 1)/a) - (log 6 + loggo)((a 1)),
(3)
13

---

Published in Transactions on Machine Learning Research (06/2025)
where
/ k2 (K²--))
) (Integera), 20²
log
Σ k=
- x exp
12
T(a+1 +
h2 -
21
log
exp
erfc
Ev(a/ (0)
r(k + 1) - k+
2x²
/20
4==
11)+1)
h22
21
log
2 exp
erfe
T(k + 1)T(( - k +
20²
22
k=
(Fractional a)..
C' is probability of client sampling. and group is sampled with probability of ming RoeT
C-MM
Compared methods: In our testing, we mainly compared: 1) the baseline method based on FedAVG and 2) DP-CLGECL. We also tested their variant versions, such as replacing AdamW with momentum.
Experiment results: The best ANLS for all E achieved by DP-CLGECI. By tuning the hyperparam- eter, the baseline method given by the competition organizers was also able to achieve higher ANLS than the baseline presented.
The ANLS of DPCLGECL was further improved by using momentum instead of AdamW, as shown in Figure A.6 This could be due the clipping radius not being well-matched with the stochastic gradient using AdamW. larger clipping radius can degrade the performance due to noise, thus, seems better to use momentum than AdamW. In this competition, mitigating the gradient drift with CLGECL was also effective in improving performance. However, calculating the stochastic gradient that matches the clipping radius was the most effective in improving performance.
5.5 Takeaways from Track 2
In the below box we highlight important takeaways from the Track
Takeaways Track 2: The participants improved the utility under DP through different approaches that are similar to the approaches in Track 1.
1. Tuning the hyperparameters in comparison to the baseline.
2. Reducing the number of updated parameters allows for more communication rounds under the same budget and enhances urivity trade-offs by decreasing sensitivity, which turn lowers the noise needed for differential privacy.
3. Changing the optimization method to avoid local gradient drifting due to data heterogeneity and DP.
Despite the different approaches improving the baseline significantly the gap between the best results of Track 2 (ANLS of 0.6225 at
8) and the non-DP results in Track (ANLS of 0.8891) remains large.
The takeaways have a strong link to prior work. Prior work recommended tuning the hyperparametors (Pono-
mareva
al.] 2023 Li et al. 2022ch and decreasing the number of parameters el. al. 2022
Tobaben
et al.
2023 Kerkouche al. 2021a b).
6 Lessons Learnt and Recommendations for Future FL and DP Competitions
In this section we present lessons learnt from organizing this competition and discuss best practices that could be considered for organizing competitions in the future.
14

---

Published in Transactions on Machine Learning Research (06/2025)
6.1 Ensuring that the Track 2 Submissions Are DP
The track 2 of this competition required participants to provide a model checkpoint trained under DP. Additionally, we asked the participants to provide privacy proof outlining how their method is formally differential private and requested the source code.
Formal privacy proof Asking for privacy proof from the participants results in two things: (1) The organizers can check that a new proposed method is DP; and (ii) The participating team can reflect on ensuring that their method is actually DP. Insufficient formal analysis in prior work has lead to response papers (Carlini et al. 2021 2022) that corrected the wrong analysis.
Ensuring that the implementations are DP While the privacy proof ensures that theorctically the submissions are DP, even small mistakes in the implementation of DP methods can invalidale severely weaken the DP guarantees (Tramer et al. 2022 Ganev et al., 2025). Among these are the clipping of the updates, the correct noise addition and scaling as well as the subsampling. Thus, members of the organizing team have inspected the implementations of the best scoring methods but this is a manual process that does not scale to competitions with a large number of participants. The code reviews could be complemented with automatic tests that increase the chance of finding bugs in the implementation. Established DP libraries such as Opacas (Youselpour al.) 2021) use unit tests but these lests are custom to the implementation that are testing and writing new tests requires much more manual labour than plain code reviews. Using only established implementations (e.g., like Opacus) for critical parts of the code would reduce the risk of bugs but also limit the possible solutions.
Automation of the validation of DP methods and implementations When scaling up the participant numbers of competition, processes need to be automated. One example for that is our automatic utility evaluation on the secret test set. Automating the validation of DP methods and implementations is less straightforward: There are methods for auditing DP implementations (Jagielski al., 2020 Nasr et al., 2023) but they are computationally expensive. Recent advancements have significantly reduced the cost of DP auditing (Steinke et al.) 2023). One option would be auditing new submissions to assist in DP validation but it is unclear how computationally costly that would be. Auditing camnot conclusively prove something DP, so it should only be used to complement privacy proofs and code checks, not replace them.
6.2 Lowering the Threshold for Participation
Referring to Table I one can see that the competition has received some interest. Also, it led to the data set being adopted in the privacy community (Wu et al. 2024) and increased the awareness in the document intelligence community (Biescas et al. 2024). Participants were required to be able to train a state-of-the- art Document Visual Question Answering model in a federated learning setting (under DP). The number of potential participants that have the required skill set is not as high as in other challenges. Thus it is important that the threshold for participation is low possible. We discuss measures that we took to lower the threshold for participation.
Starting Kit All solutions that are described in this analysis report utilized the provided starting kit to some extent. Based on the feodback from the participants, we think that the starting kit was crucial for them to participate. We can recommend to future organizers to Lest and document the starting kit. extensively and include convenience functions (e.g., to compute communication cost or DP noise).
Computational Cost Simulating the FL setting and even just fine-tuning large pre-trained models requires significant amount of compute. This is especially true under DP (Beltran et al. 2024h as the privacy/ /utility trade-off can be improved by training longer (Ponomareva et al. 2023) and using larger batch sizes Raäsä ett al. 2024). We aimed to lower the threshold for participation by reducing the size of the client datasets and utilizing not the largest pre-trained model available. Still, executing the baselines with consumer hardware is hard if not impossible. One possible avemue for the future would be to open separate tracks for consumer hardware and provide cloud compute to teams that could otherwise not participate. The recent NeurIPS 2023 challenge on LLM introduced some of these measures.
ST.LM Efficiency Challenge: ILIM+ IGIM+IGPU+DDS
github.
15

---

Published in Transactions on Machine Learning Research (06/2025)
7 Conclusion & Outlook
The challenge is a benchmark and remains open for future submissions. In the future, we will host a red team challenge at SaTML 2025, where teams run privacy attacks against models from this challenge.
Broader Impact This challenge invited the community to design novel creative solutions for real-life use cases. This has significant positive impact on users training ML models personal data. The best practices and our setup can be used to improve further challenges.
Limitations This challenge only focused on training models but does not focus on other parts of machine learning systems that may be vulnerable to privacy attacks as well (Debencdetti et al. 2024). Furthermore, we did not run any membership inference attacks, gradient inversion attacks or auditing methods to em- pirically assess the privacy leakage of the approaches but prior work (Tito et al. 2024 Pinto et al.] 2024) has shown that DocVQA models leak sensitive information about the training dataset. We do not consider settings where the data of provider is split among multiple nodes, which would require more advanced composition methods under DP and methods to detect providers that are shared among nodes. The observed settings do not focus on asyne/sync FL communication settings but assume simpler communication settings.
Author Contributions
The competition report is written vrintly by the organizers of the competition, the winners of the competition, and the runner up teams.
The organizers The organizing team consisted of Marlon Tobaben, Mohamed Ali Souibgui, Rubèn Tito, Khanh Nguyen, Raouf Kerkouche, Kangsoo Jung, Joonas Jälkö, Lei Kang, Andrey Barsky, Vincent Poulain d' d'Andecy, Aurélie JOSEPH, Josep Llados, Ernest Valveny, Antti Honkela, Mario Fritz, and Dimosthenis Karatzas. The organizers designed the challenge, provided the baseline results and ran the challenge. The organizers did not participate in the challenge. The organizers coordinated the writing of the competition report and wrote all sections in the manuscript apart from the Sections 4.3 appendices belonging to it) as they were written by the winning tcams.
4.4
5.3 and 5.4 (and the
The participants The participants do not have any connection to the organizers, and did not have access to the test data before the end of the competition. The participants were invited by the organizers to contribute to the writing of the manuscript. The participants contributions are as follows:
Section 4.3 was written by Aashiq Muhamed, Kevin Kuo, and Virginia Smith who proposed the method
that won track 1.
Section 4.4 was written by Kenta Niwa, Hiro Ishii, Yusuke Yamasaki, Takumi Fukami, Tifan Tyou, and
Rio Yokota as their proposed method was the second best entry in track 1.
Section 5.3 was written by Ragul N and Rintu Kutum that won the track 2 with their method.
Section 5.4 was written by Takumi Fukami, Yusuke Yamasaki, Kenta Niwa, and Tifan Tyou that scored the second place in track 2 with their method.
Acknowledgments
This work has been funded by the European Lighthouse on Safe and Secure AI (ELSA) from the European Union's Horizon Europe programme under grant agreement No 101070617. MT, JJ and AH have been sup- ported by the Research Council of Finland (Flagship programme: Finnish Center for Artificial Intelligence, FCAI; as well as grants 356499 and 359111) and the Strategic Research Council at the Research Council of Finland (Grant 358247). Part of this work has been performed using resources provided by the CSC IT Center for Science, Finland, and the Finnish Computing Competence Infrastructure (FCCI). MAS, RT, KN, LK, AB, JL. EV and DK have been supported by the Consolidated Rescarch Group 2021 SGR 01559 from the Research and University Department of the Catalan Government, and by project VID2023-14642611 100 funded by MCIU/AEI/ Htt//AAE/100110093 1...03039/50000 00109//51100001000 10001100011 and FSE+ RK would like to acknowledge Mphasis F1 Foundation for the computing infrastructure and financial support. Views and opinions expressed are how- ever those of the (s) only and do not necessarily reflect those of the European Union or the European Commission. Neither the European Union nor the granting authorities can be held responsible for them.
16

---

Published in Transactions on Machine Learning Research (06/2025)
References
Martín Abadi, Andy Chu, Ian J. Goodfellow, H. Brendan McMahan, Tyy Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Compuler and Communications Securily, pp. 308 318. ACM, 2016.
Srikar Appalaraju, Peng Tang, Qi Dong, Nishant Sankaran, Yichu Zhou, and R. Manmatha. Docformerv2: Local features for document understanding. In Thirty-Eightt AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sie Sirtty Conference on Innovative Applications of Artificial Intelligence, TAAT 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2024, pp. 709 718. AAAI Press, 2024.
Borja Balle, Gilles Barthe, Marco Gaboardi, Justin Hsu, and Tetsuya Sato. Hypothesis testing interpretations and renyi differential privacy. In The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, volume 108 of Proceedings of Machine Learning Research, pp. 1496-2501 PMI.R, 2020.
Sebastian Rodriguez Beltran, Marlon Tobaben, Joonas Jälkö, Niki A. Toppi, and Antti Honkela. Towards efficient and scalable training of diffcrentially private deep learning. ArXiv preprint, abs/2406.1 17298, 2024.
Daniel J. Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Titouan Parcollet, and Nicholas D. Lane. Flower: A friendly federated learning research framework. ArXiv preprint, abs/2007.14390 14390, 2020.
Nil Biescas, Carlos Boned, Josep Lladós, and Sanket Biswas. Geocontrastnet: Contrastive key-value edge lcarning for language-agnostic document understanding. In Document Analysis and Recognition ICDAR R 2024- 18th International Conference, volume 14804 of Lecture Notes in Computer Science, pp. 294-310. Springer, 2024.
Ali Furkan Biten, Ruben Tilo, Andrés Mafla, Lluís Gómez, Marçal Rusiñol, Minesh Mathew, C. V. Jawa- har, Ernest Valveny, and Dimosthenis Karatzas. ICDAR 2019 competition on scene text visual question answering. In 2019 International Conference on Document Analysis and Recognition, ICDAR 2019, pp. 1563-1570. IEEE, 2019a.
Ali Furkan Biten, Rubèn Tito, Andrés Mafla, Lluís Gómez i Bigorda, Marçal Rusiñol, C. V. Jawahar, Ernest Valveny, and Dimosthenis Karalzas. Scene text visual question answering. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, pp. 4290 4300. IEEE, 2019b.
Nicholas Carlini, Sanjam Garg, Somesh Jha, Saeed Mahloujifar, Mohammad Mahmoody, and Florian Tramèr. Neuracrypt is not private. ArXiv preprint, abs/2108.07256, 2021.
Nicholas Carlini, Vitaly Feldman, and Milad Nasr. No free lunch in "privacy for free: How does dataset condensation help privacy". ArXiv preprint, abs/2209.1499 2022.
Yannis Cattan, Christopher A. Choquette- Choo, Nicolas Papernot, and Abhradeep Thakurta. Fine-tuning with differential privacy necessitates additional hyperparameter search. ArXiv preprint, abs/2210.02156, 2022.
Christopher A. Choquette-Choo, Florian Tramer, Nicholas Carlini, and Nicolas Papernot. Label-only mem- bership inference attacks. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, volume 139 of Proceedings of Machine Learning Research, pp- 1964-1974. PMLR, 2021.
Titai Dayan, Holger R Roth, Aoxiao Zhong, Ahmed Harouni, Amileare Gentili, Anas Z Abidin, Andrew Liu, Anthony Beardsworth Costa, Bradford J Wood, Chien-Sung Tsai, et al. Federated learning for predicting clinical outcomes in patients with covid-19. Nature medicine, 27(10):1735 1743, 2021.
Soham De, Leonard Berrada, Jamie Hayes, Samuel Smith, and Borja Balle. Unlocking high-accuracy differentially private image classification through scale. ArXie preprint, abs abs/2204.13650, 2022.
Edoardo Debenedetti, Giorgio Severi, Milad Nasr, Christopher A. Choquette-Chone Choo, Matthew Jagielski, Eric Wallace, Nicholas Carlini, and Florian Tramèr. Privacy side channels in machine learning systems. In 33rd USENIX Security Symposium, USENIX Security 2024. USENIX Association, 2024.
17

---

Published in Transactions on Machine Learning Research (06/2025)
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuming of quantized 1lms. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, 2023.
Minxin Du, Xiang Yue, Sherman M. Chow, Tianhao Wang, Chenyu Huang, and Huan Sun. Dp-forward: Fino-tuning and inference on language models with differential privacy in forward pass. In Proceedings of the 2023 ACM SIGSAC Conference on Compuler and Communicalions Securily, CCS 2023, pp. 2665- 2679. ACM, 2023.
John C. Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. In COLT 2010 The 23rd Conference on Learning Theory, pp. 257-269. Omni- press, 2010.
Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sei, 9(3-4):2114400 2014.
Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, llya Mironov, and Moni Naor. Our data, our- selves: Privacy via distributed noise generation. In Advances in Cryptology EUROCRYPT 2006, 25th Annual International Conference on the Theory and Applications of Cryptographic Techniques, volume 4004 of Lecture Notes in Computer Science, pp. 486-503. Springer, 2006.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam D. Smith. Calibrating noise to sensitivity in private data analysis. J. Priv. Confidentialityy 7(3):17 51, 2016.
Ahmed Roushdy Elkordy and Amir Salman Avestimehr. Heterosag: Secure aggregation with heterogeneous quantization in federated learning. IEEE Trans. Commun., 70(4):2372 2386, 2022.
Chong Fu, Xuhong Zhang, Shouling Ji, Jinyin Chen, Jingzheng Wu, Shanqing Guo, Jun Zhou, Alex X. Liu, and Ting Wang. Label inference attacks against vertical federated learning. In 31st USENIX Security Symposium, USENIX Security 2022, pp. 1397 1414. USENIX Association, 2022.
Filippo Galli, Sayan Biswas, Kangsoo Jung, Tommaso Cucinotta, and Catuscia Palamidessi. Group privacy for personalized federated learning. In Proceedings of the 9th International Conference on Information Systems Security and Privacy, ICISSP 2023, pp. 252 263. SciTePress, 2023.
Georgi Ganev, Meenatchi Sundaram Muthu Selva Annamalai, and Emiliano De Cristofaro. The elusive pur- suit of reproducing PATE-GAN: Benchmarking, auditing, debugging. Transactions on Machine Learning Research, 2025. ISSN 2835-8856.
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients how easy is it to break privacy in federated learning? In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, 2020.
Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In Proceedings of the 35th International Conference Machine Learning, ICML 2018, volune 80 of Proceedings of Machine Learning Research, pp. 1845. PMLR, 2018
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Paramecter-cffcieen transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, volume 97 of Proceedings of Machine Learning Research, pp. 7790-2799) PMLR, 2019.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, 2022a.
Hongsheng Hu, Zoran Salcic, Lichao Sun, Gillian Dobbie, Philip S. Yu, and Xuyun Zhang. Membership inference attacks machine learning: A survey. ACM Comput. Sure., 54(11s):235:1 -235:377, 2022b.
18

---

Published in Transactions on Machine Learning Research (06/2025)
Matthew Jagielski, Jonathan R. Ullman, and Alina Oprea. Auditing differentially private machine learning: How private is private sgd? In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrlPS 2020, 2020.
Peter Kairouz, Brendan McMahan, Shuang Song, Om Thakkar, Abhradeep Thakurta, and Zheng Xu. Prac- tical and private (deep) learning without sampling or shuffling. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, volume 139 of Proceedings of Machine Learning Research, pp. 5213 5225. PMLR, 2021a.
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista A. Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D'Oliveira, Hubert Eichner, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Grutescr, Zaïd Harchaoui, Chaoyang Hc, Lic Hc, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konečný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Hang Qi, Daniel Ramage, Ramesh Raskar, Mariana Raykova, Dawn Song, Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang, Felix X. Yu, Han Yu, and Sen Zhao. Advances and open problems in federated learning. Found. Trends Mach. Learn., 14 (1-2):1 210, 2021b.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. SCAFFOLD: stochastic controlled averaging for federated learning. In Proceed- ings of the 37th International Conference on Machine Learning, ICML, 2020, volume 119 of Proceedings of Machine Learning Research, pp. 5132-5143. PMLR, 2020.
Raouf Kerkouche, Gergely Åcs, Claude Castelluccia, and Pierre Genevès. Compression boosts differentially private federated learning. In IEEE European Symposium on Security and Privacy, EuroSEP 2021, pp. 304-318. IEEE, 2021a.
Raouf Kerkouche, Gergely Ács, Claude Castelluccia, and Pierre Genevès. Constrained differentially private federated learning for low-bandwidth devices. In Proceedings of the Seventh Conference on Uncer- lainly in Artificial Intelligence, UAI 2021, volune 161 of Proceedings of Machine Leurning Research, pp. 1765-1765. AUAT Press, 2021b.
Raouf Kerkouche, Gergely Acs, and Mario Fritz. Client-specific property inference against secure aggregation in federated learning. In Proceedings of the 22nd Workshop on Privacy in the Electronic Society, WPES 2023, pp. 45-60. ACM, 2023.
Kevin Kuo, Arian Raje, Kousik Rajesh, and Virginia Smith. Sparsity for communictiinn-effcient lora. In 5th Workshop on practical ML for limited/lou resource settings, 2024.
Alexey Kurakin, Steve Chien, Shuang Song, Roxana Geambasu, Andreas Terzis, and Abhradeep Thakurta. Toward training at imagenet scale with differential privacy. ArXio preprint, Abs/2221.1232/ 2022.
Jordy Van Landeghem, Rafal Powalski, Rubèn Tito, Dawid Jurkiewiez, Matthew B. Blaschko, Lukasz Borch- mann, Mickaël Coustaty, Sien Moens, Michal Pietruszka, Bertrand Anckaert, Tomasz Stanislawek, Pawel J6ziak, and Ernest Valveny. Docunent understanding dataset and evaluation (DUDE). In ICCV, pp. 19471-19483. IEEE, 2023a.
Jordy Van Landeghem, Rubèn Tito, Lukasz Borchmann, Michal Pictruszka, Dawid Jurkicwicz, Rafal Powal- ski, Pawel Józiak, Sanket Biswas, Mickaël Coustaty, and Tomasz Stanislawek. ICDAR 2023 competition on document understanding of everything (DUDE). In ICDAR (2), volume 14188 of Lecture Notes in Computer Science, pp. 420 434. Springer, 2023b.
Kenton Lee, Mandar Joshi, Iulia Raluca Ture, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandclwal, Peter Shaw, Ming-Wo Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In International Conference on Machine Learning, pp. 18893-18914 PMLR, 2023.
19

---

Published in Transactions on Machine Learning Research (06/2025)
Vladimir I Levenshtein. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet plysics doklady, volume 10, pp. 707-710, 1966.
Junlong Li, Yiheng Xu, Tengehao Lv, Lei Cui, Cha Zhang, and Furu Wei. DiT: Self-supervised pre-training for document image transformer. In MM '22: The 30th ACM International Conference on Multimedia, pp. 3530-3330000 3539. ACM, 2022a.
Oscar Li, Jiankai Sun, Xin Yang, Weihao Gao, Hongyi Zhang, Junyuan Xie, Virginia Smith, and Chong Wang. Label leakage and protection in two-party split learning. In The Tenth International Conference on Learning Represenlations, ICLR 2022, 2022b.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEFE Signal Process. Mag. 37(3):50-00 60, 2020.
Xuechen Li, Florian Tramèr, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. In The Tenth International Conference on Learning Represenlations, ICLR 2022, 2022c.
Zheng Li and Yang Zhang. Membership leakage in label-only exposures. In CCS '21: 2021 ACM SIGSAC Conference Computer and Communiccatonss Security, pp. 880 895. ACM, 2021.
Zhuohang Li, Jiaxin Zhang, Luyang Liu, and Jian Lin. Auditing privacy defenses in federated learning via generative gradient leakage. In IEFE/CVE Conference Computer Vision and Poltern Recognition, CVPR 2022, pp. 10122 10132. LEEE, 2022d.
Guodong Long, Yue Tan, Jing Jiang, and Chengqi Zhang. Federated lcarning for open banking. In Federated Learning, volume 12500 of Lecture Noles in Compuler Science, pp. 240-254. Springer, 2020.
Virendra J. Marathe and Pallika Kanani, Subject granular differential privacy in federated learning. ArXiv preprint, ////22200006617 2022.
Minesh Mathew, Rubèn Tito, Dimosthenis Karalzas, R. Manmatha, and C. V. Jawahar. Document visual question answering challenge 2020. ArXiv preprint, /000/000.080999 2020.
Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. Docvqa: dataset for VQA on document images. In IEEE Winter Conference on Applications of Computer Vision, WACV 2021, pp. 2199 2208.
IEEE, 2021.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V. Jawahar, Infographievqa. In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022, pp. 2582-2591 IEEE, 2022.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Commmicatiinn-fffiienn learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, volume 54 of Proceed- ings of Machine Learning Research, pp. 1273 1282. PMLR, 2017a.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agũera y Arcas. Commmiicttion-cffiientt learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statislics, AISTATS 2017, volume 54 of Proceed- ings of Machine Learning Research, pp. 1273- 1282. PMLR, 2017b.
H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. In 6th International Conference on Learning Representations, ICLR 2018, 2018
Luca Mclis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Exploiting unintended feature leakage in collaborative learning. In IEFE Symposium on Security and Privacy, pp. 691-706 IEFF, 2019.
20

---

Published in Transactions on Machine Learning Research (06/2025)
Yinbin Miao, Rongpeng Xie, Xinghua Li, Ximeng Liu, Zhuo Ma, and Robert H. Deng. Compressed federated learning based on adaptive local differential privacy. In Annual Compuler Security Applicalions Conference, ACSAC 2022, pp. 159 170. ACM, 2022.
Nya Mironov. Rényi differential privacy. In 30th IEEE Computer Security Foundations Symposium, CSF 2017, pp. 263 275. IEEE Computer Society, 2017.
Nya Mironov, Kunal Talwar, and Li Zhang. Rényi differential privacy of the sampled gaussian mechanism. ArXiv preprint, abs/1908. 10530, 2019.
Mohammad Moshawrab, Mehdi Adda, Abdenour Bouzouane, Hussein Ibrahim, and Ali Raad. Reviewing federated learning aggregation algorithms; strategies, contributions, limitations and future perspectives. Electronies, 12(10))2287, 2023.
Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning: Passive and active white-box inference allacks against centralized and federated learning. In IEEE Symposium on Security and Privacy, pp. 739-77777 753. IEEE, 2019.
Milad Nasr, Jamie Hayes, Thomas Steinke, Borja Balle, Florian Tramèr, Matthew Jagielski, Nicholas Carlini, and Andreas Terzis. Tight auditing of differentially private machine learning. In 32nd USENIX Security Symposium, USENIX Security 2023, pp. 1631-1648. USENIX Association, 2023.
John Nguyen, Kshitiz Malik, Hongyuan Zhan, Ashkan Yousefpour, Mike Rabbat, Mani Malek, and Dzmitry Huba. Federated learning with buffered asynchronous aggregation. In International Conference on Artifi- cial Intelligence and Statistics, AISTATS 2022, volume 151 of Proceedings of Machine Learning Research, pp. 3581-3607. PMLR, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, pege-performance deep learning library. In Advances in Newral Information Processing Systems 32: Annuaa Conference on Neural Information Processing Systems 2019, NewrIPS 2019, pp. 8024 8035, 2019.
Balázs Pejó and Gergely Biczók. Quality inference in federated learning with secure aggregation. IEEE Trans. Big Data, 9 2023.
Francesco Pinto, Nathalie Rauschmayr, Florian Tramer, Philip Torr, and Federico Tombari. Extracting training data from document-baase based VQA models. In Forty-first International Conference on Machine Learning, ICML 2024, 2024.
Natalia Ponomareva, Sergei Vassilvitskii, Zheng Xu, Brendan McMahan, Alexey Kurakin, and Chiyaun Zhang. How to dp-fy ML: A practical tutorial to machine learning with differential privacy. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2023, pp. 5823- 5824. ACM, 2023.
Le Qi, Shangwen Lv, Hongyu Li, Jing Liu, Yu Zhang, Qiaoqiao She, Hua Wu, Haifeng Wang, and Ting Liu. Dureader vis: chinese dataset for open-domain document visual question answering. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 1338-1351. Association for Computational Linguistics, 2022.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1 140:67, 2020.
Ossi Räisä, Joonas Jälkö, and Antti Honkela. Subsampling is not magic: Why large batch sizes work for differentially private stochastic optimisation. In Forty-first International Conference on Machine Learning, TCML 2024, 2024.
21

---

Published in Transactions on Machine Learning Research (06/2025)
Arun Rajkumar and Shivani Agarwal. A differentially private stochastic gradient descent algorithm for multiparty classification. In Proceedings of the Fifteenth Internalional Conference on Artificial Intelligence and Statistics, AISTATS 2012, volume 22 of JMLR Proceedings, pp. 933 941, JMLR.org, 2012.
Vipula Rawte, Amit P. Sheth, and Amitava Das. A survey of hallucination in large foundation models. Xrx preprinl, abs/ 230990359922 2023.
Reza Shokri and Vitaly Shmatikov. on Communication, Control, and Computing, Allerton 2015, pp. 909-910. IEEE, 2015.
deep learning. In 53rd Annual Allerton Conference
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In IEFE Symposium on Security and Privacy, pp. 3-18. TEEE Computer Society, 2017.
Hira Shahzadi Sikandar, Huda Waheed, Sibgha Tahir, Saif UR. Malik, and Waqas Rafique. A detailed survey federated learning attacks and defensos. Electronics, 12(2):260, 2023.
Stěpán Simsa, Milan Sule, Michal Ufičár, Yash Patel, Ahmed Hamdi, Matëj Kocián, Matyáš Skalický, Jiff Matas, Antoine Doucet, Mickaël Coustaty, et al. Docile benchmark for document information localization and extraction. In International Conference Document Analysis and Recognition, pp. 147-166. Springer, 2023.
Jinhyun So, Corey J. Nolet, Chien-Sheng Yang, Songze Li, Qian Yu, Ramy E. Ali, Basak Guler, and Salman Avestimehr. Lightsecagg: a lightweight and versatile design for secure aggregation in federated learning. In MLSys, 2022.
Shuang Song, Kamalika Chaudhuri, and Anand D. Sarwate. Stochastic gradient descent with differentially private updates. In IEEE Global Conference on Signal and Information Processing, GlobolSTP 2013, pp- 245 248. IEEE, 2013.
Thomas Steinke, Milad Nasr, and Matthew Jagielski. Privacy auditing with one (1) training run. In Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Sys- tems 2023, NewrIPS 2023, 2023.
Anshuman Suri, Pallika Kanani, Virendra J. Marathe, and Daniel W. Peterson. Subject membership infer- ence attacks in federated learning. ArXiu preprint, abs/2206.00 03317, 2022.
Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. Visualmre: Machine reading comprehension on document images. In Thirty-Fifte AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty- Third Conference on Innovative Applications of Artificial Intelligence, TAAT 2021, The Fleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, pp. 13878 13888. AAAI Press, 2021.
Hanlin Tang, Chen Yu, Xiangru Lian, Tong Zhang, and Ji Liu. Doublesqueeze: Parallel stochastic gradient descent with double-pass error-compensated compression. In Proceedings of the 36th International Con- ference on Machine Learning, 1CML 2019, volume 97 of Proceedings of Machine Learning Research, pp. 6155-6165. PMLR, 2019.
ICDAR (2), volume 12822 of Lecture Notes in Computer Science, pp. 778 792. Springer, 2021a. Rubèn Tito, Dimosthenis Karatzas, and Ernest Valveny. Document collection visual question answering. Tn
Rubèn Tito, Minesh Mathew, C. V. Jawahar, Ernest Valveny, and Dimosthenis Karatzas. ICDAR 2021 competition document visual question answering. In TCDAR (4), volume 12824 of Lecture Notes in Computer Science, pp. 635 649, Springer, 2021b.
Rubèn Tito, Dimosthenis Karatzas, and Ernest Valveny. Hierarchical multimodal transformers for multipage docvqa. Pallern Recognit.. 144:109834, 2023.
22

---

Published in Transactions on Machine Learning Research (06/2025)
Rubèn Tito, Khanh Nguyen, Marlon Tobaben, Raouf Kerkouche, Mohamed Ali Souibgui, Kangsoo Jung, Joonas Jälkö, Vincent Poulain D' D'Andecy, Aurélie Joseph, Lei Kang, Ernest Valveny, Antti Honkela, Mario Fritz, and Dimosthenis Karatzas. Privacy-aware document visual question answering. In Document Anal- ysis and Recognition - ICDAR 2024- 18th International Conference, volume 14809 of Lecture Notes in Computer Science, pp. 199 218. Springer, 2024.
Marlon Tobaben, Aliaksandra Shysheya, John Bronskill, Andrew Paverd, Shruti Tople, Sanliago Zanella Béguelin, Richard E. Turner, and Antti Honkela. On the efficacy of differentially private few-shot image classification. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
Florian Tramèr, Andreas Terzis, Thomas Steinke, Shuang Song, Matthew Jagielski, and Nicholas Carlini. Debugging differential privacy: A case study for privacy auditing, ArX preprint, abs/2202. 2022.
Florian Tramèr, Gautam Kamath, and Nicholas Carlini. Position: Considcrations for differentially private learning with largo-scale public pretraining. In Forty-first International Conference on Machine Learning, TCML 2024, 2024.
Stacey Truex, Nathalie Baracaldo, Ali Anwar, Thomas Steinke, Heiko Ludwig, Rui Zhang, and Yi Zhou. A hybrid approach to privacy-preserving federated learning- In Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security, AISec@CCS 2019, pp. 1-11. ACM, 2019.
lifan Tyou, Tomoya Murata, Takumi Fukami, Yuki Takezawa, and Kenta Niwa. A localized primal-dual method for centralized/ decentralized centralized/dccnntrriizdd federated learning robust to data heterogencity. IEEE Trans. Signal Tnf. Process. over Networks, 10:94-105 2024.
Aidmar Wainakh, Fabrizio Ventola, Till Müßig, Jens Keim, Carlos Garcia Cordero, Ephraim Zimmer, Tim Grube, Kristian Kersting, and Max Mühlhäuser. User-level label leakage from gradients in federated learning. Proc. Priv. Enhancing Technol. 2022(2):227-2444 244, 2022.
Tong Wu, Ashwince Panda, Jiachen T. Wang, and Prateck Mittal. Privacy-preserving in-context learning for large language models. In The Twelfth International Conference Learning Represenlations, ICLR 2024, 2024.
Jinjin Xu, Wenli Du, Yaochu Jin, Wangli He, and Ran Cheng. Ternary compression for communication- efficient federated learning. IEFE Trans. Neural Networks Learn. Syst., 33(3):1162-1176; 2022.
Prateek Yadav, Leshem Choshen, Colin Raffel, and Mohit Bansal. Compeft: Compression for communicating parameter efficient updates via sparsification and quantization. ArXie preprint, abs/2311.13171, 2023.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advonces in Neural Information Processing Systems 27: Annual Conference on Newral Information Processing Systems 2014, pp. 3320 03000-32,..0 2014.
Yeojoon Youn, Zihao Hu, Juba Ziani, and Jacob D. Abernethy. Randomized quantization is all you need for differential privacy in federated learning. Ar preprinl, abs/2306. 11913, 2023.
Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, Graham Cormode, and llya Mironov. Opa- cus: User-friendly differential privacy library in Py Torch. ArXiv preprint, hts/2109.122000 2021.
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A. Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manocl, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. Dif- ferentially private fine-tuning of language models. In The Tenth International Conference on Learning Representations, ICLR 2022, 2022.
Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, 2020.
23

---

Published in Transactions on Machine Learning Research (06/2025)
Xiaojin Zhang, Yan Kang, Kai Chen, Lixin Fan, and Qiang Yang. Trading off privacy, utility, and efficiency in federated learning. ACM Trans. Intell. Syst. Technol, 14(6))98:119988) -9::32, 2023.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. idlg: Improved deep leakage from gradients. ArXiv preprint, (bs/2001.0022 2020.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. In Advances in Neural Information Processing Systems 32: Anmual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, pp. 14747-14756, 2019.
24

---

Published in Transactions on Machine Learning Research (06/2025)
A General Appendix
A.1 Dataset
This section contains additional information regarding the dataset. The data set is described in more detail in Tito et al. (2024) and is available to download. The Dataset is based on images from the DocILE dataset (Simsa el. al., 2023), which was published under the MIT License, but has new annotations for these images. The new annotations are the OCR transcriptions (using Amazon Textract) and the pairs of duestion/answerr The dataset has been published under the Licence CC-BY-4.0
Unseen Providers
6574
Train data
6573
Valid data
6572
Test data
client 9
6050
6049
Provider
6048
1002
client
1001
1000
client 0
20
100
Document
Figure A.1: Data split of the PFL-DocVQA dataset.
Dataset
Client (Subset)
Provider
Document
Page
Question/ Answer
400
2224
5930
19465
2382
22229
2
404
2296
6667
21673
3
414
2358
6751
22148
4
429
4543
12071
32472
Train
5
423
2378
22361
6
423
2700
7406
23801
7
416
1951
5617
18462
8
401
1932
5421
17868
2136
6353
20840
Valid
2231
3536
9150
30491
In-Distribution
1390
2875
25603
Test
Out-of-Distte Distributio
977
1912
5375
17988
Table A1:
Statistics on
the basc
PFL-DocVQA
Dataset
in terms of number of
Providers/ Documents/ Paaass/aa///aaa watpa///aaaaaaeaetonnnn--aaae Quee//uuuuunonseerr
25

---

Published in Transactions on Machine Learning Research (06/2025)
Cllent 0
Client
107
Providers
201
100
100
125
25
100
125
150
Client
Client
10a
Previderrs
207
30/-
30"
50
100
125
150
25
100
125
150
Client
Client
10a
trovideers
301-
50
75
100
125
150
25
50
75
100
125
150
Client
Client
Povddes
301
30
100
10a
0
25
50
75
100
125
150
25
50
75
100
125
150
Documents
DDocuments
Client
Client
10a
10F
Povidrss
101-
10
100
10a
0
25
50
73
100
125
150
25
50
75
100
125
150
Documents
Documents
Figure A.2: Distribution of documents provider split by client in the used version of the PFL-DocVQA dataset. Note that two providers are missing as they are siguificant outliers: In client 6 there is one with 531 documents and in client 4 one with 2283 documents.
26

---

Published in Transactions on Machine Learning Research (06/2025)
A.2
Additional information on the model
The pre-trained model (Tito et al. 2023) can be found at https://hugginggace..o/// http://aaacccoouunnnntttttttooot vt5-base-saa
spdosvaaa It is licensed under the gpl-3.0 license.
A.3
Training details for baselines
The hyperparameters for the baseline were chosen using a combination of grid search and manual search.
The assumption for the basclines is not to have optimal hyperparameters but rather reasonable baselines.
We utilize two NVIDIA A40 (40 GB VRAM each) and train for some hours to obtain the baselines. The exact runtime depends on the hyperparamters being used.
B Privacy Analysis of baseline track
The privacy analysis our differentially private baseline discussed in this section. The provided python script to compute the privacy budget e is derived from the following analysis.
B.1 Definitions
Definition B.1 (Differential Privacy Dwork Roth (2014)). randomized mechanism M with range satisfies (e, diffferrenta privacy, if for any two adjacent datasets E and E", ie., E' EU {a} for some æ in the data domain (or vice versa), and for any subset of outputs O S R, it holds that
Pr(M(E) E < ef Pr(M(E") E 0
(A1)
Intuitively, DP guarantees that an adversary, provided with the output of M, can draw almost the same conclusions (up to s with probability larger than 1 - 6) about any group no matter if it included in the input of M or not Dwork Roth (2014). This means, for any group owner, privacy breach is unlikely be due to its participation in the dataset.
In Federated Learning, the notion of adjacent (neighboring) datasets used in DP generally refers to pairs of datasets differing by one client (client-level DP), or by one group of one user (group-level DP), or by one data point of one user (record-level DP). Our challenge focuses on the group-level DP Galli et al. (2023) where each group refers to a provider.
We use the Gaussian mechanism to upper bound privacy leakage when transmitting information from clients the server.
Definition B.2. (Gaussian Mechanism Dwork & Roth (0014) Let f R" - Rd be an arbitrary function
that maps n-dimensional input to d logits with sensitivity being:
S ( E.E
(A2)
over all adjacent datasets E and E. The Gaussian Mechanism Ma, parameterized by o, adds noise into the output, i.c.,
[()) = ( +1(00021).
(A3)
As in Abadi et al. (0166) Mironov et al. (2019) we consider the Sampled Gaussian Mechanism (SGM) -a composition of subsampling and the additive Gaussian noise (defined in B.5 for privacy amplification. Moreover, we first compute the SGM's Renyi Differential Privacy as Mironov et al. (2019) and then we
use conversion Thcorem B.8 from Balle al. (020 for switching back Differential Privacy.
Definition B.3 (Rényi divergence). Let and Q two distributions on X' defined over the same probability space, and let p and q be their respective densities. The Rényi divergence of finite order a + between P and Q is defined follows:
27

---

Published in Transactions on Machine Learning Research (06/2025)
/ g(x))
(((x) g(x:)
Da Q a
dr.
Rényi divergence at orders 00 defined by continuity.
Definition B.4 (Rényi differential privacy (RDP)). A randomized mechanism M: & - R satisfies (a, p)- Rényi differential privacy (RDP) if for any two adjacent inputs E, E' E & it holds that
Da (M(E) M(E)) SP
In this work, we call two datasets E, E' to be adjacent if E' = E {Ø} (or vice versa).
Definition B.5 (Sampled Gaussian Mechanism (SGM)). Let. f be an arbitrary function mapping subsets of E to Rd. We define the Sampled Gaussian mechanism (SGM) parametrized with the sampling rate S 1 and the noise > 0 as
SG - f (fx: [ sampled with probability 4)) + +N,0 o²I"),
where each element of E is independently and randomly sampled with probability without replacement.
As for the Gaussian Mechanism, the sampled Gaussian mechanism consists of adding i.id Gaussian noise with zero mean and variance a² each coordinate value of the true output of f. In fact, the sampled Gaussian mechanism draws vector values from multivariate spherical (or isotropic) Gaussian distribution which described by random variable (O0 (20)) where d is omitted if is unambiguous in the given context.
B.2 Analysis
The privacy guarantee of EL-GROUP-DP is quantified using the revisited moment accountant Mironov et al. (2019) that restates the moments accountant introduced in Abadi et al. (2016) using the notion of Rényi differential privacy (RDP) defined in Mironov (2017).
Let No denote the pdf of o2), and let μi denote the pdf of ((L, Let be the mixture of two Gaussians μ = (1 - 4)Ho + gP1, where q is the sampling probability of single record in single round.
Theorem B.6. Mironov et al. 2019). Let SG be the Sampled Gaussian mechanism for some function for any adjacent E, E' E Then SG solisfies (o.p)-RDDP if
J and under the assumplion 2 S
5 - log (maxxAaa>a Ba)
(A4)
where A. A Eerwo (o(e)o)((())/)(())) and B. A E=>uu(u((()/u(()1)
Theorem B.6 states that applying SGM to a function of sensitivity (Equation B.2 at most 1 (which also holds for larger values without loss of generality) satisfies (a, p),P)-RDP if p - log Amax(AA, B.)). Thus, analyzing RDP propertics of SGM equivalent to upper bounding A, and B..
From Corollary Mironov el al. 2019 An Ba for any a 2 Therefore, we can reformulate A4 as
S Ev(a)g) = log A,
(A5)
To compute Ass we use the numerically stable computation approach proposed in Mironov al. (2019) (Sec. 3.3) depending whether is expressed as integer real value.
28

---

Published in Transactions on Machine Learning Research (06/2025)
Theorem B.7 (Composabilityy Mironov (0017 Suppose that mechanism M consists of sequence of adaptive mechanisms Mi,...,MR where M; II RJ R, If all the mechanisms in the sequence are PR-D)) then the composition of the sequence is (a, RDD.
In particular, Theorem B.7 holds when the mechanisms themselves are chosen based on the (public) output of the previous mechanisms. By Theorem B.7 it suffices to compute {v((ag) at each step and sum them up to bound the overall RDP privacy budget of an iterative mechanism composed of single DP mechanisms at each step.
Theorem B.8 (Conversion from RDP to DP Balle et al. 2020 If mechanism M is (a, P)-DP then il is ((p + log((a- - 1)/a) (log + log a) / 1),6) DP for any 0 < 6 < 1.
29

---

Published in Transactions on Machine Learning Research (06/2025)
C Details on Track 1
Algorithm Update rules of FodShampoo
1: P Initialization wi, Lg,b = R,, = PL = PR
2: for C. (I,..., R) (Outer loop round) do
3: Global model update in central
4: Averaging of aggregated local models
úi k -i=1 ")
5: 'Iransmit global model to clients
Transmit,erver-clien
6: > (ii) Local model updates in each client T: for v ~ [V] (K = 2 client sampling) do & Initialization of local model
8:
+ 10
9:
for 1. E (l...., T (Inner loop iteration) do
10:
è Local stochastic gradient E Ed
11:
for be (I...., B) (Layer-wise iteration) do
12:
Reshaping elements of regarding b-th layer to be matrix form G..b Bhrn,*doout..
mod(t, 10)
then
Local update of preconditioning matrices using moving average
end L,, L,,b + G. R.D R,, + [G,,) G.,b
mod(t, 100)
Computing local preconditioning matrices
T., - [T,, + 11 114
RS RR. 1
18:
end if
19:
P Local model update using wiee clipping
W Wi C.ppLL,,GeRRR......
end for end for
Reshaping model in matrix form into w; <- Vec(WW....... W.,
'Transmit local model to central server
Transmitcleass--eeet -server (w)
24: end for
23: end for
30

---

Published in Transactions on Machine Learning Research (06/2025)
validation loss
validatioo
validation anis
0.500
3.95
3,90
0.99
3.85
0,026
0.024
0.750
0.022
3.70
0.755
0.020
3.855
0.270
3880
Communication cost [MIB]
Communication cost [MIB]
Communication COST [MIB]
Adimik 9.0 9999 Rower
Sharm pon ze d 10 9, 989) Lune Step., 128, CPPondiennce (-0.15,0155)
Loves Step. Cipuustennce
Shar
Lusur Stop,. 128, Cupuraccem (-02,02)
11 g- Ci
Step,. 128, C.ppanntiomn
Figure A.3: Hyperparameter tuning for FedShampoo
Test loss
Test acc
anis
0.800
0.895
0.990
0.815
0.026
0.785
0.024
0.875
0.700
0.870
0.725
0.020
10
15
20
30
35
60
5
10
20
25
30
40
45
10
15
30
36
Communication cost [MiB]
Communication cost [M B]
Communication cost [M]3]
Figure A.4: Convergence curves for the global model using (Left) validation loss, (Center) validation accuracy, and (Right) validation ANT.S.
31

---

Published in Transactions on Machine Learning Research (06/2025)
D Details on Track 2
Algorithm 2 Update rules of DP-CI.GECL
Server:
Initialize common model wo for 8 = to R do
Select set K of clients randomly 5: for each client k in K do
6:
" = (wientx(w;-))
7: end for
Wy = 1 + 1 Σx o
E
9: end for
10: Output: Global model wy
11: Clienty (w:-1):
12: G& is set of predefined disjoint groups of records in D₁
13: Select M C G2 randomly
14: if == 1 then
15: Randomly initialize Ao
16: else
17: A--1 < de-2 + W - w/-2
18: end if
19: for each group G in M. do
20: w' W4-1
21: Au = AdamW(G, w/, Tya) W2-1 + A
22: Aud = w// max(1, [|u 12]
23: end for
24: w/ = W [M]
(Eo As +N((,.00'))
25: Output: Client model wi- W1-1
32

---

Published in Transactions on Machine Learning Research (06/2025)
(C)
0.60
0.58
0.58
0.58
0.56
0.56
0.56
ANLS
0.54
ANLS
0.54
0.52
0.52
0.52
0.50
0.50
0.48
0.48
0.48
0.46
0.46
0.46
0.44
12
14
2
10
12
14
10 12 14
Communication rounds [timses]
Communication round [timses]
Communication round [timses]
DP-CLGECL Baseline (w hyperparameter tuning)
Baseline (w/o hyperparameter tuning)
Figure A.5: convergence curve evaluating using the global model. (a) validation ANLS at E = 1, (b) validation ANLS at 4, (c) validation ANT.S at & 8. We used clipping radius $ 0.5, the number of client sampling C 2, the learning rate 7 0.0002, and the number of communication round Rt 14 for hyperparameter selection.
0.60
0.56
ANLS
0.54
0.52
0.50
0.48
0.46
0.44
0
2
10 12
14
Communication rounds[times]
DP-CLGECL
Baselines (w hyperparameter tuning)
DP-CLGECL with momentum
Baseline with momentum(w hyperparameter tuning)
Figure A.6: convergence curve evaluating using the global model at e = 1. (Teft) Validation accuracy, (right) Validation ANLS. We used clipping radius S = 0.5, the number of client sampling C 2, learning rate 7 0.0004, and the number of communication round 12.
33


Reviews:
[{"review_id": "f79KVf25a2", "reviewer": "Reviewer_71WV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* Starter kit was provided for the participants to understand the implementation details \n* The problem of automated invoice processing in a federated setting is interesting\n\n\nWeaknesses:\n\n* It is simply a benchmark paper. I do not think TMLR is a suitable venue for the same.\n* FL nodes do not have invoices from common providers, which limits scope significantly \n* All techniques discussed are prior works. \n* No model inversion attack seems to have been considered to demonstrate the leakage aspect in practice\n* How do the organizers ensure that the submitted code is correct is not clear. Furthermore, there is not much attempt to address the DP analysis verification.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript discusses the various aspects of the Privacy Preserving Federated Learning Document VQA (PFL-DocVQA) competition of NeurIPS 2023. The goal of the competition was to develop comprehensive solutions for federated invoice processing that are provably private and communication-efficient. Essentially, the task was to extract information and reason over real invoice document images based on provided questions and answers, while considering the privacy-preserving scenario where the data is distributed across multiple nodes in a federated learning setting. Therefore, the PFL-DocVQA brought together people from many domains, including document analysis, privacy, and federated learning.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Provide clear justification on why datasets and benchmarks should be publishable in TMLR\n* Provide the model inversion attacks results", "broader_impact_concerns": "I do not have broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bPvteDv9VX", "reviewer": "Reviewer_GHaR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength \n\n$\\bullet$ The authors have presented the efficiency gains from each of the ML approaches and algorithmic optimization, along with a detailed ablation study in Appendix. \n\n$\\bullet$ The authors have done a great job of explaining the difference between the two competition tracks (communication-efficient FL and differentially private learning),  making it clear why each one matters and what challenges they address. \n\n\n$\\bullet$ The breakdown of efficiency gains and optimization techniques used for high-efficiency gains without sacrificing the utility (in DP-settings), making it a useful resource for future research and further exploration of privacy-preserving methods in real-world settings.\n\n\n### Weaknesses \n\n$\\bullet$ The biggest problem with this paper is that it’s essentially just a competition summary with no real research contribution. It doesn’t introduce any new methods or offer any deeper insights beyond the obvious takeaways from the competition results. The benefits of LoRA fine-tuning for efficiency and quantization for communication reduction are already well understood, so the paper doesn’t add anything meaningful, and seems more like a report than a research paper.\n\n\n$\\bullet$ The current version of the manuscript is unnecessarily short at nine pages, with important ablation studies pushed to the appendix instead of being included in the main text, making it harder to get a complete picture of the trade-offs involved in different techniques. \n\n\n$\\bullet$ The paper is difficult to follow, even for experts, due to its disorganized structure and lack of a clear narrative. Key takeaways are not well-highlighted, making it hard to quickly grasp the main contributions. Instead, information is scattered across dense sections, requiring readers to sift through lengthy explanations to extract relevant insights. A more structured presentation with clearer sectioning and emphasis on key findings would greatly improve readability.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents an overview of the NeurIPS'23 competition on privacy-preserving federated learning (FL) for document visual question answering (VQA), highlighting the ML techniques (e.g., LoRA) and optimization strategies (FedShampoo) used by the winning and runner-up teams across the competition’s two tracks: communication-efficient FL and differentially private learning.  The authors have clearly shown the efficiency gains from each of the optimization steps employed by the winners and runners for both the tracks, which could serve as a useful reference for future research looking to optimize the trade-offs between privacy, utility, and communication in similar settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weaknesses.", "broader_impact_concerns": "It could have been better-articulated.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zy1tBP5fji", "reviewer": "Reviewer_NWHm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a very unusual work - I am actually struggling to identify any area from the potential contributions list from the TMLR submission website, so in my view this work does not necessarily fall into the scope of TMLR. Moreover, it is not immediately clear to me which specific part of this work was done by the authors and by the submitted works during the challenge. The references to 'we' are somewhat misleading unless everything was done by the authors (which judging by the context is not the case?). Nonetheless, I will leave the decision on the suitability to the AE and will provide the objective/actionable feedback on the work itself.\n\nThe problem that the work studies is very relevant and important for both the fields of FL and document-based VQA. However the actual analysis of the problem, approaches and the results are in my view on a shallow side. I additionally had a hard time understanding the work itself - the structure is convoluted, there is no easy way to distinguish what the work ACTUALLY proposed. There are some tasks to solve, solutions, takeaways, but these all seem to be completed by different people (since participants provided the solutions, not the authors from my understanding, but correct me if I am wrong). So in short: I do not actually understand what specifically this manuscript is about: lessons learnt from the challenge? Or how the challenge was approached? Or new promising direction for VQA in FL? There is a little bit on all of these topics here, but none of them are addressed in an adequate level of details.\n\nContent-wise part 2 (differential privacy) demonstrates the issues I have with the manuscript very well. Your background on DP is very limited: you jump straight into eps-delta DP without ANY explanation of the principles behind DP, pure-DP definition, or any background which is actually relevant to the definition (admittedly some of it is in the appendix, but there is no reference to the appendix in your background section at all). 'High utility models' under DP seems to be a semi-random collection of DP references, it lacks structure and relevance to the problem. I would advise the authors to rework DP section to a) have a concrete structure the reader can actually follow, b) establish concrete relevance to the problem at hand, because just referring the reader to other papers for both the relevant literature and the details on DP is not acceptable. There is no discussion on the relevance of DP-SGD, its issues in FL, any alternatives such as DP-FTRL - this has to be addressed. Your entire second part of the submission is based on DP, otherwise it is challenging to understand both the setting of the problem as well as what are the existing challenges which could not have previously been solved using standard approaches.\n\nMoreover, when we actually look into the analysis you for some reason claim epsilon of 1.0 is 'medium'? Medium with respect to what? Is there any relevant literature which would call this epsilon level 'medium'? But more interestingly authors later claim that  'The baseline solution for track 2 utilizes DP stochastic optimization.' What exactly does this mean? Are you referring to DP-SGD? Because the context suggest this, but there is no specific mention otherwise. Appendix seems to refer to the established DP theorems, proofs and definitions, not to anything specifically tailored to the problem at hand up until the very end, when a new previously undiscussed definition of DP for FL appears out of nowhere. Lessons learnt with respect to DP are also not particularly novel or useful: 'needing to ensure that implementations are DP' is absolutely trivial. Asking for proofs? You already stated in the problem definition that this is required, why is this a lesson learnt at the end? No FL-specific issues are mentioned at all as well: subsampling, sync/async FL wrt DP etc. These would have been very interesting topics for discussion, but they are simply missing.\n\nSo in essence the DP section requires a SUBSTANTIAL rework with respect to having a clear and concise literature review at the start with all relevant DP definitions; methods for DP training of ML models (including those relevant to LMs like DP-Forward and those relevant to FL like DP-FTRL etc.), novel actionable insights on how to interpret DP section of the results etc. \n\nSection on FL suffers from similar issues: not enough details on the solutions proposed, what are the new novel insights that can be gained etc. We already know that using LoRA can limit the communication cost and lead to better models, for instance. More subjectively I would have liked to have seen more details on the challenge itself (without having to look it up): how is the data distributed, is async FL tested at all etc. There is no clear setup-solution-message in this section either, but it is maybe severe compared to DP.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work present the solutions to the previously held NeurIPS challenge on privacy-preserving federated document VQA. The challenge is split into two (distinct, but complementary) parts: minimal required utility threshold and differentially private learning. This work goes through the challenge itself, the solutions (with background) and lessons learnt.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "As above, there is no specific structure to the manuscript to describe what the contribution is. In my view the work itself requires a serious rewrite based on what specifically is the message the authors want to convey.", "broader_impact_concerns": "I am not sure on the impact of this work at all, I struggle to see what exactly is the contribution here (and how this contribution is split between the participants of the challenge and the authors of the manuscript). I am not in favour of accepting this work because a) I do not see how this is relevant to the list of topics for TMLR submissions and b) I struggle to see the useful contributions that belong to the authors here.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers raised issues on the authorship as the authors wrote about winners and participants' solutions and ablations as part of the contribution of the paper. During discussion between AE and authors, authors provided information that participants and winners are actually co-authors of the paper, but this cannot be revealed due to anonymity. However, I ask authors to make a clear contributions list below in the "Requested changes" to make sure authorship is properly provided.

All reviewers agreed that the updated revision is more solid as a research paper and now articulates the message and story better. However, all of the concerns of Reviewer 71WV were articulated only as limitations, which may not be the best course of action. Part of the concerns I think is okay to keep as limitations, but some could be improved for the final revision (see "Requested changes").

The last critical issue raised by Reviewer NWHm is the helpful / interesting takeaways from the paper as the must thing for TMLR papers. While the Review agreed that readability is improved, despite the authorship issue, the Reviewer still doesn't see interesting / new / helpful takeaways. As I stated in the "Audience" section, takeaways are helpful from an empirical perspective from my point of view even having all the things which are known. However, to improve this message even more, I ask authors to expand a bit on the takeaways to make connections and refer to whether observations are in agreement or not with prior results.

**Requested changes:**
- As the paper is co-authored by both organizers and participants at the same time, and also some organizers were participants (correct if I am wrong, but also this is very biased to me as organizers had access to the closed set, thus maybe this should be prohibited by rules that participants can be from organizers' list), authors must de-anonymize the paper including all participants' names and provide their exact contributions / methods they used. It is probably good to specify in the author list exactly who are organizers, who are participants co-authoring the paper, and then provide detailed contributions in the end of the paper.
- Expand more details regarding questions raised by Reviewer 71WV. Particularly
  - "No model inversion attack seems to have been considered to demonstrate the leakage aspect in practice" -> could authors provide some references to justify that this is the real issue and leakage happens in practice (kind of obvious thing to me, but good to support with prior work reported this issue)
  - "How do the organizers ensure that the submitted code is correct is not clear. Furthermore, there is not much attempt to address the DP analysis verification." -> could authors put more details into the text how exactly organizers were double checking code and proofs (maybe running code, maybe doing code review, maybe re-implementing, etc.) - this is a key thing of the paper to share practices + raise discussion on how we could check things in the critical domains where error cost is huge and harmful.
- Expand key takeaways to make connections to currently observed empirical results (to fully address concern from Reviewer NWHm): showcase what is aligned and what is not, what is something new (if any) which is not probed before (e.g. some hyperparameters are very crucial to make things work in practice despite theory).

**Note: If changes will not be integrated I would like to retract "acceptance with minor revision" and change status to "rejection with major revision" as authorship and contributions from participants must be properly reflected.**
Decision: Accept (poster)