Package: adaswarm
AdaSwarmThis repo purportedly implementsAdaSwarm, an optimizer, that combines Gradient Descent and Particle Swarms.AdaSwarmis based on "AdaSwarm: Augmenting Gradient-Based optimizers in Deep Learning with Swarm Intelligence,Rohan Mohapatra, Snehanshu Saha, Carlos A. Coello Coello, Anwesh Bhattacharya Soma S. Dhavala, and Sriparna Saha", to appear in IEEE Transactions on Emerging Topics in Computational Intelligence. An arXiv version can be foundhere.Thisrepo contains implementation used the paper.WhyAdaSwarm:Said  et  al.[1]postulated  that  swarms behavior is similar to  that of classical  and  quantum  particles.  In  fact, their analogy is so striking that one may think that the social and  individual  intelligence  components  in  Swarms  are,  after  all, nice useful metaphors, and that there is a neat underlying dynamical system at play. This dynamical system perspective was indeed useful in unifying two almost parallel streams, namely, optimization  and  Markov  Chain  Monte  Carlo  sampling.In a seminal paper, Wellington and Teh[2], showed that a  stochastic  gradient  descent  (SGD)  optimization  technique can  be  turned  into  a  sampling  technique  by  just  adding noise,  governed  by  Langevin  dynamics.  Recently,  Soma  and Sato[3]provided further insights into this connection based on  an  underlying  dynamical  system  governed  by  stochastic differential equations (SDEs).While these results are new, the connections  between  derivative-free  optimization  techniques based on Stochastic Approximation and Finite Differences are well documented[4]. Such strong connections between these seemingly  different  subfields  of  optimization  and  sampling made  us  wonder:  Is  there  a  larger,  more  general  template, of which  the  aforementioned  approaches  are  special  cases, exist?AdaSwarmis a result of that deliberation.We believe that it is just a beginning of a new breed ofcomposable optimizersWhat isAdaSwarmin simple terms, in the context Deep Learning:Setupy: responsesf(.)is a model specified by a network with parameterswf(x)is the prediction at observed featurexL(.)is loss, to drive the optimizationApproximate gradients ofL(.)w.r.tf(.)run an independent Swarm optimizer overL(.)with particle dimension equal to the size of the network's output layerusing swarm particle parameters, approximate the gradient ofL(.)w.r.tf(.)Get gradients off(.)w.r.twusing standardAutoDiff, via chain rule, get the approximate gradient off(.)w.r.twApproximate gradients ofL(.)w.r.twvia Chain Ruletake the product of gradients in steps (2) and (3)Updates the network weights via standard Back PropagationWhy does it work? Minor changes are meaningful!At this time, we could embellish the fact that Swarm, by being a meta-heuristic algorithm, has less tendency to get trapped in local minimal or has better exploration capabilities. It is helping the problem overall. Secondly, entire information about the "learning" the task comes from the loss, and the functionf(.)only specifies the structural relationship between input and output. Moreover, the ability of EMPSO, the first step toward AdaSwarm facilitates exploration and exploitation equally by using a modified formulation leveraging exponentially averaged velocities and by not ignoring past velocities. It is these velocities (which are different at different stages in the search space) that make the difference at local minima successfully by being able to differentiate between stagnating regions/saddle points and true local minima. Particular objects of interest are the equivalence theorems, such as the following:"partial derivative of the loss wrt the weights" can be expressed in terms of Swarm parameters thus keeping tight control over the hyper-parameters and not tuning those at all for convergence. This addresses a common complaint about meta-heuristics.So, having better "optimization" capabilities at the loss, in general, are going to be helpful. While we have ample empirical evidence that shows thatAdaSwarmis working well, we also have some theory (not complete but enough to offer the convergence insights, particularly from the point of robust loss functions such as MAE and irregular losses used to solve PDEs/PDEs such as the Schrodinger Equation).Another speculation, speculation at this time, is that, to the best of our knowledge, all current optimization techniques only harvest information coming from a single paradigm.AdaSwarm, whereas, combines different perspectives, like in an ensemble. More than an ensemble, it is a composition -- where different perspectives get chained. That is one fundamental difference betweenAdaSwarmand other population-based techniques.In someways, just like an neural network architecture is composed of several layers,AdaSwarmis a composition of optmizers. That composition eventually fits into the chain rule.As a result, the changes are very small. Same is the case with Adam and RMSProp, right? Other notable examples, where we see pronounced differences in speed/convergence, with very simple changes in the maps are:Proximal gradient descentvsAccelerated Proximal gradient descentEulervsLeapFrog_ ...Therefore, in order to better understand, and develop the theory and tools for composable optimizers, we have to develop both theoretical and computational tools to understand why and whereAdaSwarmworks. Along the way, make such optimizers accessible to the community.Adaswarm Equivalence of Gradients: Why is it happening?The equivalences are driven by the following equations in the main text (cf. docs/papers folder):Eqns (4)-(6), (15), (18) and the eqn below for non-differentiable loss and (20)-(24)Objectives:Develop a plug-and-play optimizer that works withother optimizers in the PyTorch ecosystem, along side the likes ofAdam,RMSProp,SGDany architectureany datasetwith the same api as others, i.e.,optim.AdaSwarm()Battle test on variety oftest objectives functionsdatasetsarchitectures (Transformers, MLPs,..)losses (BCE, CCE, MSE, MAE, CheckLoss,...)paradigms (RIL, Active Learning, Supervised Learning etc..)etc..Provide insights into the workings ofAdaSwarmbyanalyzing the workings of the optimizersvisualizing the path trajectoriesetc..Most importantly, be community driventhere is lot of enthusiasm and interest in the recent graduates and undergraduates, that want to learn ML/AI technologies. Instead of fiddling with MNIST datasets, and predicting Cats and Dogs, do something foundational and meaningful. If you take offence to statement, you are not ready for this project.turn this into a truly community-driven effort to offer a useable, useful, foundational building block to the deep learning ecosystemHow to run this projectPre-requisitesPython 3.9+PoetryGet the source$ git clone https://github.com/AdaSwarm/AdaSwarm.gitRunning the example (single method - runs Adaswarm/PSO)$ cd AdaSwarm
$ make runRunning tests$ make testUsing in your own projectIf you use poetry you can add AdaSwarm as a dependency$ poetry add adaswarmSee.example/main.pyfor usage.Contributing:While we are yet to establish the policy to contribute, we will follow how any Apache open source project works. For example, see airflow project'scontributionguidelines.[tbd]developer slack channel will be coming soonwe will hold weekly zoom/meet at specific times (off office hours, so anybody can join)But be mindful. There may not be any no short-term rewards.Research is bloody hard work. There will not be any instant gratification or recognition for the work. Expect lot of negative results, and set backs.Optimization problems are generally hard, and writing an Engineering level framework that works onanyproblem is even harder. It is scientific computing, not writing hello world examples.So take a plunge only if you are willing to endure the pain, w/o worrying about the end result.References[1]S. M. Mikki and A. A. Kishk, Particle Swarm Optimizaton: A Physics-Based Approach.    Morgan & Claypool, 2008.[2]M.  Welling  and  Y.  W.  Teh,  “Bayesian  learning  via  stochastic  gradient langevin dynamics,”In Proceedings of the 28th International Conference on Machine Learning, p. 681–688, 2011.[3]S.  Yokoi  and  I.  Sato,  “Bayesian  interpretation  of  SGD  as  Ito  process,” ArXiv, vol. abs/1911.09011, 201.[4]J.  Spall, Introduction  to  stochastic  search  and  optimization. Wiley-Interscience, 2003CitationAdaSwarmwill be appearing in thepaperyou can cite:@inproceedings{adaswarm,title="daSwarm: Augmenting Gradient-Based optimizers in Deep Learning with Swarm Intelligence",author="Rohan Mohapatra, Snehanshu Saha, Carlos A. Coello Coello, Anwesh Bhattacharya Soma S. Dhavala, and Sriparna Saha",booktitle="IEEE Transaction on Emerging Topics in Computational Intelligence",month=tbd,year="2021",address="Online",publisher="IEEE",url="https://arxiv.org/abs/2006.09875",pages=tbd}