Title: Regret Analysis of RMSProp and AdamNC for Training Deep Interpolating Neural Networks
Primary area: optimization
Keywords: regret analysis, root mean square propagation, deep neural network, online learning

Abstract: We provide a theoretical analysis for RMSProp and AdamNC (Adam without corrective terms) for training deep fully-connected neural networks with smooth activations, in the online learning setting. We focus on the binary classification tasks with logistic loss or exponential loss. We assume that the model can interpolate data, i.e., it can obtain an arbitrarily small loss $\varepsilon$, while the distance to the initialization is bounded by a decreasing function $g(\varepsilon)$. We show that the regret is upper bounded by $\mathcal{O}(\text{poly} [g(1/T )])$ provided that the width is at least $\mathcal{O}(\text{poly} [g(1/T )])$, where $T$ is the total iteration number. We further show that under NTK-separability, the regret is less than $\mathcal{O}(\text{poly}(\log T))$ when the width is larger than $\mathcal{O}(\text{poly}(\log T))$. We also provide a comparable regret bound for the scalar version of RMSProp and AdamNC, without requiring prior knowledge of problem parameters for learning rates. Our analysis can also be applied to smooth losses, leading to similar regret bounds.