Title: Riemannian Stochastic Weakly Convex Optimization Under Heavy-Tailed Noises
Primary area: optimization
Keywords: Stochastic optimization, Heavy-Tailed noises, Riemannian optimization, weakly convex, Moreau envelope

Abstract: Many studies have focused on stochastic optimization methods in Euclidean space under heavy-tailed gradient noises. Recently, some research has observed that gradient noise on Riemannian manifolds may also exhibit heavy-tailed distribution. In this paper, we investigate two types of gradient noise: sub-Weibull-type noise and noise with a bounded $p$-th central moment ($p$-BCM) for $p \in (1,2]$. The latter is more challenging due to the possibility of infinite variance when $p \in (1,2]$. Under both gradient noise assumptions, high-probability convergence of stochastic optimization methods in Euclidean spaces has been extensively studied in the contexts of convex (nonconvex) and smooth (nonsmooth) optimization. However, our understanding of the high-probability convergence of stochastic optimization methods on Riemannian manifolds under these two types of noise remains incomplete. We study the high-probability convergence of the ordinary Riemannian stochastic subgradient descent (SsGD) on the Stiefel manifold under weak convex optimization with sub-Weibull-type noise, as well as the convergence of Riemannian clipped-SsGD under $p$-BCM type noise. For weakly convex objective functions that may be nonconvex and nonsmooth, our results show that, compared to the Euclidean case, the theoretical dependence on failure probability and iteration complexity of Riemannian SsGD under sub-Weibull-type noise is not degraded. Under $p$-BCM noise, our findings indicate that the non-smoothness and non-convexity of weakly convex objectives do not affect the theoretical dependence of Riemannian clipped-SGD on failure probability.