Dale meets Langevin: A Multiplicative Denoising Diffusion Model
Authors: name: Nishanth Shetty, name: Madhava Prasath, name: Chandra Sekhar Seelamantula
Organization: Indian Institute of Science

Gradient descent has proven to be a powerful and effective technique for
optimization in numerous machine learning applications. Recent advances in
computational neuroscience have shown that learning in standard gradient
descent optimization formulation is not consistent with learning in biological
systems. This has opened up interesting avenues for building biologically
inspired learning techniques. One such approach is inspired by Dale's law,
which states that inhibitory and excitatory synapses do not swap roles during
the course of learning. The resulting exponential gradient descent optimization
scheme leads to log-normally distributed synaptic weights. Interestingly, the
density that satisfies the Fokker-Planck equation corresponding to the
stochastic differential equation (SDE) with geometric Brownian motion (GBM) is
the log-normal density. Leveraging this connection, we start with the SDE
governing geometric Brownian motion, and show that discretizing the
corresponding reverse-time SDE yields a multiplicative update rule, which
surprisingly, coincides with the sampling equivalent of the exponential
gradient descent update founded on Dale's law. Furthermore, we propose a new
formalism for multiplicative denoising score-matching, subsuming the loss
function proposed by Hyvaerinen for non-negative data. Indeed, log-normally
distributed data is positive and the proposed score-matching formalism turns
out to be a natural fit. This allows for training of score-based models for
image data and results in a novel multiplicative update scheme for sample
generation starting from a log-normal density. Experimental results on MNIST,
Fashion MNIST, and Kuzushiji datasets demonstrate generative capability of the
new scheme. To the best of our knowledge, this is the first instance of a
biologically inspired generative model employing multiplicative updates,
founded on geometric Brownian motion.