Our work in this area is both analytical and synthetic. We are interested in describing the stochastic dynamics of these algorithms, and using the analyses to improve the algorithms' utility. Our research in dynamics of on-line learning led to the synthesis of a stochastic approximation algorithm that effectively uses cost function curvature information and provides nearly theoretically optimal asymptotic convergence rates. Our algorithm uses an adaptive momentum sheme that achieves the optimal convergence rates asymptotically, but requires only linear time and storage requirements . Here's a synopsis .
Several papers take up the use of master equation techniques to analyze on-line learning algorithms.
At this point of the evolution, the density is broadly spread out through the low-cost regions with two prominent spikes in density that correspond to local optima. The analysis through the master equation is well suited to study the algorithms' asymptotic behavior including the effects of learning rate annealing, and algorithmic attempts to obtain optimal asymptotic convergence rates. Our work in stochastic approximation theory includes two innovations that go beyond the diffusion equations we and other researches in the field explored. This remains, to my knowledge, the only known exact solution to the master equation for a stochastic approximation algorithm. The expansion gives solutions in increasing powers of the learning rate.