Title: Non-differentiable Regularization for Heavy-tailed Differentially Private Stochastic Convex Optimization
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Non-differentiable regularization, heavy-tailed gradients, differential privacy, stochastic convex optimization

Abstract: Recently, differentially private stochastic convex optimization (DP-SCO) with heavy-tailed (HT) (sub)gradients has attracted increasing attention. Weaker than the uniform Lipschitz continuity assumption, the HT assumption on the stochastic (sub)gradients aligns more closely with real-world data distributions. However, few existing methods can handle the non-differentiable regularization (NDR) for HT DP-SCO. The main difficulty lies in achieving a low excess population loss and a low computational complexity simultaneously. In this work, we propose a novel forward-backward splitting approach to tackle NDR for HT DP-SCO, abbreviated by NDR-HT. It satisfies concentrated differential privacy, achieves an asymptotically optimal excess risk bound (up to logarithmic factors), and requires only $\mathcal{O}(n \log n)$ gradient evaluations, which is a lower computational complexity than those of existing state-of-the-art approaches. Furthermore, NDR-HT achieves linear convergence up to an additive approximation error and avoids solving complex subproblems in each iteration. Extensive experiments on both synthetic and real-world data show that our approach is effective and efficient in solving HT DP-SCO with NDR.

TL;DR: We propose a novel forward-backward solving approach to tackle non-differentiable regularization for  heavy-tailed differentially private stochastic convex optimization.