Title: Stochastic Gaussian Zeroth-Order Optimization: Improved Convergence Analysis under Skewed Hessian Spectra
Primary area: optimization
Keywords: stochastic zeroth-order optimization, quadratic regularity, skewed Hessian spectra

Abstract: This paper addresses large-scale finite-sum optimization problems, which are particularly prevalent in the big data era. 
In the field of zeroth-order optimization, stochastic methods have become essential tools. 
Natural zeroth-order stochastic methods primarily rely on stochastic gradient descent (SGD).
Preprocessing the stochastic gradient using a Gaussian vector defines the method ZO-SGD-Gauss (ZSG), whereas estimating coordinate-wise partial derivatives defines ZO-SGD-Coordinate (ZSC).
Compared to ZSC, ZSG often demonstrates superior performance in practice.
However, the underlying mechanisms behind this phenomenon remain unclear in the academic community.
To the best of our knowledge, our work is the first to theoretically analyze the potential advantages of ZSG compared to ZSC.
To facilitate convergence analysis, the quadratic regularity assumption is introduced to generalize the smoothness and strong convexity to the Hessian matrix.
This assumption makes it possible to integrate Hessian information into the complexity analysis.
In the quadratic case, we provide a theoretical analysis and proof of the significant convergence improvement achieved by ZSG. Finally, experiments on both synthetic and real-world datasets validate the effectiveness of our theoretical analysis.