Title: Private Zeroth-Order Optimization with Public Data
Venue: NeurIPS 2025
Accepted: yes
Keywords: Zeroth-Order Optimization

Differential Privacy
Abstract: One of the major bottlenecks for deploying popular first-order differentially private (DP) machine learning algorithms (e.g., DP-SGD) lies in their high computation and memory cost, despite the existence of optimized implementations.  Zeroth-order methods have promise in mitigating the overhead, as they leverage function evaluations to approximate the gradients, hence significantly easier to privatize. While recent works have explored zeroth-order approaches in both private and non-private settings,  they still suffer from relatively low utilities compared with DP-SGD, and have only been evaluated in limited application domains. 
In this work, we propose to leverage public information to guide and improve gradient approximation of private zeroth-order algorithms. We explore a suite of \underline{p}ublic-data-\underline{a}ssisted \underline{z}eroth-\underline{o}rder optimizers (PAZO) with minimal overhead. We provide theoretical analyses of the PAZO framework under an assumption of the similarity between public and private data. Empirically, we demonstrate that PAZO achieves superior privacy/utility tradeoffs across vision and text tasks in both pre-training and fine-tuning settings, outperforming the best first-order baselines (with public data) especially in highly private regimes, while offering up to $16\times$ runtime speedup.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to accelerate zeroth-order differentially private optimization while improving gradient approximation by leveraging public data. The authors propose three PAZO variants and validate them both theoretically and empirically.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths \n- Proposed three versions of the optimization method to utilize public data in zeroth-order settings.\n- The proposed methods often achieve performance comparable to first-order DP methods that use public data, while significantly improve the speed of private learning.\n- The privacy-utility trade-off is mitigated, which implies that PAZO can be useful where $\\epsilon$ is small.\n\n## Weaknesses\n- PAZO improves performance by using public data under the γ-similarity assumption. However, this raises the question: isn't the improvement mostly due to the public data alone? That is, isn’t it possible that the private data contributed very little during training? In fact (1) The performance does not vary much with the privacy budget. (2) The results are not very sensitive to hyperparameters.\nIn first-order methods, private gradients are clipped, whereas public gradients are not, meaning public gradients may dominate the update. As a result, private gradients might have a negligible impact. To address this, the authors should conduct an experiment using only public data (i.e., no private data, hence no privacy concerns), and explicitly report how much performance improves when private data is included.\n\n- In the vision domain, the authors assume in-distribution public data, while in the language domain, they assume out-of-distribution (OOD) public data. This seems to be done to satisfy the γ-similarity assumption. Perhaps in language tasks, even OOD data still lies in a similar feature space. If that is the case, the paper should discuss domain-specific similarity more clearly or unify the assumption across domains. If in-distribution public data is assumed, this should be clearly stated at the beginning of the paper.\n\n\n### Minor weaknesses\n- Notation should be changed. The notation of the model is $x$, while that of the data is $\\xi$. Normally, the use of the notation $x$ refers to data and $\\theta$ or $w$ to parameters. \n- Comparisons between methods using public data and those that do not (e.g., Figures 2 and 5) are unclear. As done in Tables 1–4, figures should visually differentiate methods that use public data (e.g., using distinct colors or styles).\n\nQUESTIONS:\n- Why is the noise added to the scalar directional derivative $\\{f(x+\\lambda u)- f(x -\\lambda u)\\} / 2\\lambda$, instead of $\\text{clip}(\\{f(x+\\lambda u)- f(x -\\lambda u)\\} / 2\\lambda) \\cdot u$ as DPZero[7]? DPZero added noise to the vector after clipping, resulting in the noise added in all $d$ dimensions. However, in this paper, scalar noise is added and then multiplied by the random vector. This makes the direction of $u$ preserved. What are the privacy implications of preserving the direction of $u$? Even though the vector $u$ is random, it is used as an input in the directional derivative $f(x+\\lambda u)- f(x -\\lambda u) / 2\\lambda$, and thus may potentially leak private information. How can this be justified from a privacy perspective?\n- Figure 4 only reports PAZO-S. However, the convergence theorems show different upper bounds for PAZO-M, P, and S. Based on these, we would expect different convergence behaviors. Can the authors provide convergence plots for each variant?\n\nLIMITATIONS:\nMore interpretation of experimental results is needed regarding robustness and the privacy-utility tradeoff.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a novel zero-order optimization algorithm that is guided by some first-order information (i.e., gradients) of publicly available data. They introduce three different ways of integrating such information in the zero-order optimization process and show that they can achieve faster convergence while still keeping the privacy advantages coming with zero-order optimization.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The idea is interesting and reasonable in distributed environments where data privacy is important\n- the paper is written clearly\n- a convergence analysis is provided\n- empirical results look promising\n\n**Weaknesses**\n\n- in Algo. 1 it is unclear how the batch of public data is obtained. Since it should match with the private batch as good as possible, the authors should describe how they ensure that both batches exhibit similar data characteristics (if they exhibit conflicting characteristics, divergence might even happen)\n- in the unnumbered between 141-142 it is not clear to me why with the presented equation, the coefficients $u$ are learned. Please clarify this.\n- in the Equation before line 142, why is it $\\mathbb{E}_u[\\nabla f(x)^T GuGu]$? Isn't it only $Gu$? If so, the subsequent equations are also wrong. A brief derivation would be helpful here.\n- The authors claim an equivalence between their projection of the gradient and the expectation of gradient approximations when sampling from a ball induced by $G$ (see lines 143-144). Please provide proof of this equivalence.\n\nSince I am not familiar with the field of DP and zero-order optimization, my review should be used with care for the final decision.\n\nQUESTIONS:\n- wouldn't a repeated querying of the same function lead to a much higher privacy budget needed to ensure DP?\n\nLIMITATIONS:\nThe work should explicilty discuss current limitations (at least briefly).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers to use public data to assist private zeroth-order optimization. Existing works have shown that zeroth-order methods can be applied to differentially private training or fine-tuning. However, there is still a gap in terms of performance compared with first-order methods. This work proposes a suite of public data assisted zeroth-order optimizers (PAZO), including PAZO-{M,P,S}. PAZO-M simply uses a convex combination of private zeroth-order estimations and public gradient as the descent step. PAZO-P narrows the search space of zeroth-order optimization into a subspace constructed using public gradient. PAZO-S uses private data to select the best public gradient. Convergence analysis shows that all three methods match the worst-case upper bound of existing results. Empirical experiments show that all three methods match or even outperform exising public data assisted first-order methods, while offering speedup in time.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is clearly written, and the studied topic is interesting and important for the community. The claims are supported by evidence, and the results look good to me. My concerns are following.\n\n1. Why is $\\gamma$ similarity used? A more natural measure to me is the cosine similarity. In all convergence results, there is dependence on $\\sigma$. I understand the meaning of $\\sigma_1$ and $\\sigma_2$, but what is $\\sigma$? What is the dependence on $q$ and the dimension $d$ in all theoretical results? The dependence on the dimension $d$ is crucial in zeroth-order methods and should be highlighted and well discussed.\n\n2. In several places of the paper, the authors say existing zeroth-order methods only consider prompt tuning for language models, like [6,7,8,10]. I think this is not a correct statement. Prompt tuning is one efficient way of doing language model fine-tuning, where only prompt embeddings are updated and the base model is frozen. This is not what [6,7,8,10] do. In these related works, all trainable model parameters are updated and thus considered to be full parameter fine-tuning. The authors should correct all these wrong statements.\n\n3. One of the major reasons that previous works propose to use zeroth-order methods is to save memory by avoid gradient computations. In PAZO, public gradients are anyway required, and thus this benefit of saving memory is lost. Is there a comparison in terms of GPU memory for all methods, including vanilla DPZero and DPSGD, PAZO-{M,P,S}, and public-data assisted first-order methods?\n\n4. The authors claim in the contribution that improved convergence can be guaranteed using a proper choice of parameters. It is not clear in the main text how the rates can be improved. Is this improvement only in terms of constants or also in terms of key parameters like $T$ and $d$?\n\nQUESTIONS:\nMajor questions are already asked in weakness. Other minor:\n\n1. Typos: should it be \"rely on\" instead of \"reply on\" in line 21 and line 132? I guess it should be \"combined with these prior methods\" in line 88.\n\n2. The paper uses random direction uniformly on the sphere when introducing the algorithm. Random search directions from Gaussian are also extensively used in previous work. Which noise is used in the experiments? How do these two compare in both the theory and experiments?\n\n3. Isn't it costly to sample $k$ mini-batches in PAZO-P and PAZO-S? What is the value of $k$ and batch size $b$ used for each method when comparing time efficiency in Figure 5?\n\nLIMITATIONS:\nLimitations are already discussed in the weakness, including dimension dependence in theoretical results and memory consumption.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors proposed combining public first-order gradients with private zeroth-order gradients for differentially private training.  The paper proposes three ways of mixing the gradients: 1. naive weighted averaging; 2. projecting ZO direction to FO gradient subspace; 3.   using zeroth-order evaluation to determine the optimal public gradient direction. The paper provides both convergence and privacy analyses of the proposed algorithms, and presents numerical results on CV and NLP datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The paper provides novel ways of mixing zeroth and first-order gradients for differentially private optimization.\n2. The paper introduces $\\gamma$-similarity to measure the dissimilarity between the public and private data distributions.\n3. The paper provides a theoretical analysis for the privacy and convergence guarantee of all three proposed algorithms under reasonable assumptions.\n4. Numerical results show that the proposed algorithms outperform the baseline algorithm DPZero and are less sensitive to $\\epsilon$.\n\nWeakness:\n1. For PAZO-P and PAZO-S, how does the memory consumption compare with DP-Zero or PAZO-M? A key benefit of using ZOO is its memory-saving. If the FO gradient matrix G uses too much memory, there might be no point in using the ZO gradient. \n2. For PAZO-M, clipping the function value itself does not make sense to me. In the case that $f$ is very large (or very small), clipping it to C does not provide any information. It might be more reasonable to clip the function difference. \n3. Is argmin the best operator for PAZO-S? The author fails to investigate other smoothed versions of argmin to avoid directly dropping the other gradient information.\n\n4. For theoretical analysis, the convergence results rely on the clipping being inactive. While in DPZero, the effect of clipping is taken into consideration with the concentration bound. Therefore, the analysis requires further refinement to avoid assuming the clipping is inactive and the bounded gradient Assumption 1.\n5. The paper does not provide the final privacy-utility trade-off, i.e., the optimal choice of steps $T$ for convergence. \n6. In the proof in B.4 for PAZO-S, in lines 550 and 551, it is unclear to me why the expected value of $g_t - g'(x_t;B'_{*})$ is bounded by $\\sigma^2_2$. Since after argmin, $g'$ is no longer unbiased.\n\n7. I didn't find the choice of $b$ in Table 8. Also, for DP-SGD, why is $b$ not an integer?\n8. Since the paper introduced $\\gamma$, it is good to have a set of experiments demonstrating how the algorithm performs with different $\\gamma$'s for one dataset.\n\nQUESTIONS:\nPlease address the weakness.\n\nI believe the theoretical analysis is limited and may contain errors. If my concerns about the theoretical analysis can be addressed, I would love to increase my score.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}