Title: PREAMBLE: Private and Efficient Aggregation via Block Sparse Vectors
Venue: NeurIPS 2025
Accepted: yes
Keywords: Differential Privacy

Vector Aggregation

Two-server model
Abstract: We revisit the problem of secure aggregation of high-dimensional vectors in a two-server system such as Prio. These systems are typically used to aggregate vectors such as gradients in private federated learning, where the aggregate itself is protected via noise addition to ensure differential privacy. Existing approaches require communication scaling with the dimensionality, and thus limit the dimensionality of vectors one can efficiently process in this setup. 

We propose PREAMBLE: {\bf Pr}ivate {\bf E}fficient {\bf A}ggregation {\bf M}echanism via  {\bf BL}ock-sparse {\bf E}uclidean Vectors. PREAMBLE builds on an extension of distributed point functions that enables communication- and computation-efficient aggregation of {\em block-sparse vectors}, which are sparse vectors where the non-zero entries occur in a small number of clusters of consecutive coordinates. We show that these block-sparse DPFs can be combined with random sampling and privacy amplification by sampling results, to allow asymptotically optimal privacy-utility trade-offs for vector aggregation, at a fraction of the communication cost. When coupled with recent advances in numerical privacy accounting, our approach incurs a negligible overhead in noise variance, compared to the Gaussian mechanism used with Prio.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces **block-sparseness** as a principled abstraction for private aggregation, striking a balance between accuracy and efficiency. The authors extend distributed point function (DPF) constructions to support *$k$-block-sparse vector secret sharing*, achieving significantly reduced communication and computation overhead—transmitting only \\$kB\\$ field elements instead of full sparse vectors. They integrate their scheme with *sampling-based privacy accounting* and demonstrate that it achieves privacy-utility trade-offs *comparable to the Gaussian mechanism* while drastically reducing communication costs. For example, on 8-million dimensional data, their method compresses communication from 64MB to 1MB with only a modest increase in noise for $(1, 10^{-6})$-DP.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. **Principled Use of Block-Sparsity**:\n   The paper introduces *block-sparsity* as a natural and effective abstraction for high-dimensional vector aggregation, striking a balance between expressiveness and efficiency. This insight enables meaningful communication and computation reductions without compromising accuracy.\n\n2. **Communication Efficiency**:\n   By extending DPF constructions to handle block-sparse vectors, the authors reduce communication costs from *tens of megabytes (e.g., 64MB)* to as low as *~1MB*, making the approach highly practical for large-scale systems.\n\n3. **Improved Server-Side Efficiency**:\n   The protocol minimizes server workload by leveraging *cuckoo hashing* and *probabilistic batch codes*, reducing server-side PRG evaluations from $O(mD)$ to $O(D)$, a significant scalability improvement.\n\n4. **Compatible with DP Accounting and Utility Guarantees**:\n   The authors show that their block-sparse aggregation integrates well with *privacy amplification by subsampling* and *numerical DP accounting*, leading to utility comparable to the Gaussian mechanism, despite the structural constraints.\n\n5. **Efficient Zero-Knowledge Proofs**:\n   The paper includes a *zero-knowledge proof system* for validating block-sparse commitments, which does not inflate asymptotic runtime or communication cost, a valuable feature for ensuring integrity in adversarial environments.\n\n6. **Deployment-Oriented Design**:\n   The use of counter-mode PRG evaluations and linear or near-linear runtime makes the construction well-suited for *practical, scalable deployment*, especially in two-party and federated learning settings.\n\nWeaknesses:\n\n1. **Lack of Some Definitions**:\nClearly, this paper must have been written by one or more crypto experts. PRG (assuming it stands for Pseudorandom Generator) is referenced a few times without an actual definition. See line 118 where it first appears...\n\n2. **Implementation Complexity**:\n   The protocol’s reliance on advanced cryptographic components (e.g., DPFs, cuckoo hashing, batch codes, ZKPs) may introduce *engineering complexity* that could hinder adoption without reference implementations.\n\nQUESTIONS:\n* Can you comment on how sensitive your protocol's performance is to violations of the block-sparsity assumption? \n\n* Are there heuristics or preprocessing techniques to induce block structure when it is not present?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method for efficient and private aggregation of high-dimensional vectors in the two-server model. They introduce a new cryptographic primitive based on block-sparse distributed point functions to reduce communication and computational overhead. By leveraging block sparsity, random sampling, and privacy amplification by sampling, PREAMBLE achieves strong privacy guarantees with communication costs significantly lower than prior work. The method is particularly relevant in settings like private FL, where clients must send large gradient vectors.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n+ The paper is well-written and easy to read. Ideas and analysis have been clearly presented.  \n+ The paper addresses a practical bottleneck in private FL, high communication costs for secure aggregation. \n+ The abstraction of k-block sparse vectors and their integration into DPFs is innovative and leads to notable efficiency gains. The authors have also combined formal privacy guarantees with empirical utility assessments. \n\nWeaknesses\n\n- While training machine learning models with differential privacy is inherently challenging, the evaluation in this paper is limited to relatively small-scale tasks, which may not fully demonstrate the method’s scalability or practical applicability. The evaluated models are relatively small, which may limit the generalizability of the results to more complex architectures commonly used in practical federated learning scenarios. Does the conclusion change when we have a more complicated task, e.g., a large model on ImagNet level classification?\n\n- While communication is reduced, the paper may potentially downplay the increased client and server-side computational cost due to PRG evaluations and zero-knowledge proofs. A detailed breakdown of computational costs would be helpful.\n\n- Though many baselines are mentioned, it lacks a direct comparison with recent alternatives like [CSOK23]. It would be helpful to include more results and discussion.\n\nQUESTIONS:\n- Could you provide insights or preliminary discussions on how PREAMBLE performs with larger models (e.g., ResNet or ViT) and more complex datasets?\n\n- Could you provide a detailed breakdown of runtime or resource usage for both the client and the servers? \n\n- Could you provide direct comparison with recent alternatives like [CSOK23]?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of private aggregation of high-dimensional vectors. In this problem, there is a data set of high-dimensional vectors distributed among some large number of clients, and there are two servers whose goal is to compute and output a private estimate of their sum. The main goal of this paper is to reduce the communication cost of protocols that achieve this. One approach from prior work is to randomly sparsify and scale the vectors so that in expectation they are preserved, and then to communicate the lightweight sparse vector. The authors state that this approach still leads to a communication cost of $m \\lambda \\log D$, where $D$ is the dimensionality of the vectors, $m$ is the number of coordinates subsampled during sparsification, and $\\lambda$ is a security parameter used in secret sharing.\n\nThe approach taken by the authors is to use block-sparsity, i.e. instead of randomly picking some $m$ coordinates to sample, the vector is divided into blocks of size $m/k$ and some $k$ of these blocks are sampled and picked for communication. By doing so they state that they get communication cost $O(m + k\\lambda \\log D)$, potentially with some other side-costs which can be dealt with separately.\n\nFurther, since the final goal is differentially private aggregation, the authors show that one of the advantages of their method is that by using block sparsity and privacy amplification by subsampling, the potentially high sensitivity of the subsampled and scaled vector can be offset leading to a reasonable privacy utility trade-off.\n\nIn order to construct their protocol that requires the destination servers to aggregate and privatize the sparsified and encrypted vectors, the authors rely upon zero knowledge proofs to ensure that the clients can prove that their vectors are indeed $k$-block sparse.\n\nFinally, some experiments are conducted showing that they are indeed able to reduce communications costs and truncation error with blocking, and that it works well with privacy by conducting a numerical privacy accounting.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The problem being considered is interesting and impactful.\n2. The high-level approach makes sense and the gains made seem to be significant.\n\nWeaknesses:\n1. Although the paper is well-written, it is hard for someone not familiar with Prio to easily gauge correctness, or understand all the roadblocks being encountered and how they are resolved.\n2. The collaborative computation between the servers for the private output is not clear from the pseudocode.\n\nQUESTIONS:\n1. I understand that this is a hybrid trust model of sorts - the communication between the clients and the two servers is cryptographically secure, and the output of the servers is privacy-preserving in the sense of differential privacy. At the outset you mention how it is only required that one of the two servers is assumed to be honest. If a server is malicious but receives some secret shares of the sparsified vector, what exactly is the privacy promise that is achieved here? Is only the final output private or has the malicious server gained some private information?\n2. In your Overview section you give an example of a secret share using Distributed Point Functions that seems to use 15MB of communication. Is it easy to see what your approach would give here? It might be good to place this in that section for an easy comparison.\n3. What exactly are the Zero Knowledge Proofs needed for, is this proof of sparsity necessary for reasons of privacy? Also what is the total round complexity of your protocol?\n4. Would the gains made from privacy amplification by subsampling also extend to the prior work which uses randomly sparsified vectors, potentially offsetting the high worst case sensitivity?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the private communication of vectors from each client to an aggregating server under differential privacy, while taking communication cost into account. In particular, the authors propose an extension from 1-block sparse vectors to k-block sparse vectors. Each client divides their vector into B blocks, randomly selects k of them, and sends only those to the server. Assuming each block has a bounded ℓ₂ norm, they show that the DP release of the k-block sparse vector can approximate the sum of all client vectors, up to an error term. By appropriately choosing parameters, they demonstrate that the noise from DP-SGD will dominate this approximation error. The authors also provide theoretical privacy guarantees, which depend on a parameter c, although its construction or approximation is not fully understood. They validate their theoretical results experimentally on toy examples and compare DP-SGD with the block-sparse vector approach using numerical privacy accounting.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper addresses an important and timely problem in differentially private (DP) communication.\n- The theoretical results support the proposed approach for DP k-sparse communication. The explanations of the sparsification algorithm, hashing, and decoding mechanisms are clear and well-presented.\n\nWeaknesses:\n\n\n\n-  In the numerical comparisons with DP-SGD, the authors mention that both algorithms operate under the same privacy budget (via numerical accounting), but they do not specify the values of ε and δ. Moreover, there is no discussion of how the results are impacted in lower-privacy regimes — the ones typically desired in practice.\n\n- The privacy guarantee relies on a parameter -c- that can be arbitrarily small. While it's theoretically appealing to have a guarantee without approximation, the practical value of such a guarantee is questionable.\n\n- While some parts of the paper provide thoughtful discussions of the results, other sections merely restate the findings in a verbal form without offering deeper insights or analysis e.g. after theorem 3.1 when it is explained in what scenario the noise from DP takes over without any explanation of whether such scenario is reasonable other than not.\n\nQUESTIONS:\n- How can we approximate the constant c?\n- Can you perform an ablation study with varying levels of Gaussian noise added, specifically in the low and medium privacy regimes and if so what would be the utility gap between DP SGD and the proposed work?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-09"}