Title: Eliminating Steady-State Oscillations in Distributed Optimization and Learning via Adaptive Stepsize
Primary area: learning on graphs and other geometries & topologies
Keywords: Distributed learning, linear convergence, adaptive stepsize

Abstract: Distributed stochastic optimization and learning is gaining increasing traction due to its ability to enable large-scale data processing and model training across multiple agents without the need for centralized coordination. However, existing distributed stochastic optimization and learning approaches, such as distributed SGD and their variants, generally face a dilemma in stepsize selection: a small stepsize leads to low convergence speed, whereas a large stepsize often incurs pronounced steady-state oscillations, which prevents the algorithm from achieving stable convergence accuracy. In this paper, we propose an adaptive stepsize approach for distributed stochastic optimization and learning that can eliminate steady-state oscillations and ensure fast convergence. Such guarantees are unattained by existing adaptive stepsize approaches, even in centralized optimization and learning. We prove that our proposed algorithm achieves linear convergence with respect to the iteration number, and that the convergence error decays sublinearly with the batch size of sampled data points. In the specific case in terms of deterministic distributed optimization with exact gradients accessible to agents, we prove that our proposed algorithm linearly converges to an exact optimal solution. Moreover, we quantify that the computational complexity of the proposed algorithm is on the order of $\mathcal{O}(\log(\epsilon^{-1}))$, which matches the existing results on adaptive stepsize approaches for centralized optimization and learning. Experimental results on machine learning benchmarks confirm the effectiveness of our proposed approach.

TL;DR: We propose an adaptive stepsize approach that  eliminates steady-state oscillations and ensures fast convergence in distributed  learning and optimization.