# Conditions for mathematical equivalence of Stochastic Gradient Descent and Natural Selection

_Many thanks to Peter Barnett, my alpha interlocutor for the first version of the proof presented, and draft reader._

Analogies are often drawn between natural selection and gradient descent (and other training procedures for parametric algorithms). It is important to understand to what extent these are useful and applicable analogies.

Here, under some modest (but ultimately approximating) simplifying assumptions, natural selection is found to be mathematically equivalent to an implementation of stochastic gradient descent.

The simplifying assumptions are presented first, followed by a proof of equivalence. Finally, a first attempt is made to consider the effect of relaxing each of these assumptions and what departure it causes from the equivalence, and an alternative set of assumptions which retrieves a similar equivalence is presented.

## Summary of simplifying assumptions

It is essential to understand that the equivalence rests on some simplifying assumptions, none of which is wholly true in real natural selection.

1. Fixed 'fitness function' or objective function mapping genome to continuous 'fitness score'
2. Continuous fixed-dimensional genome
3. Radially symmetric mutation probability density
4. Limit case to infinitesimal mutation
5. A degenerate population of 1 or 2
6. No recombination or horizontal transfer

(NB assumptions 5 and 6 will be addressed in detail and amended later)

## Proof

### Setup and assumptions

Let us make some substantial modelling simplifications, while retaining the spirit of natural selection, to yield an 'annealing style' selection process.

We have continuous genome $\mathbf x_t \in \Bbb R^n$ with fixed fitness/objective function $f : \Bbb R^n \to \Bbb R$

Mutations are distributed according to probability density $P_m : \Bbb R^n \to [0, 1]$ which is _radially symmetric_

- so $\forall \mathbf \epsilon \in \Bbb R^n. P_m(\mathbf \epsilon) = g(|\mathbf \epsilon|)$ for some $g : \Bbb R^+ \to [0, 1]$ (a density, but not a probability density)
- a mutation in one direction is just as likely as a mutation in another direction

We also consider the case of mutations _very small_ relative to the genome, tending to infinitesimal.

Selection is stochastic, but monotonically determined by the fitness differential, according to selection function $P_s : \Bbb R \to [0, 1]$

- so the probability of $\mathbf x'$ being selected over $\mathbf x$ is $P_s(f(\mathbf x') - f(\mathbf x))$
- $P_s$ is a monotonically nondecreasing function: a greater 'fitness score' differential can not reduce chances of selection
  - i.e. $\forall \Delta, \delta \in \Bbb R. \Delta > \delta \implies P_s(\Delta) \ge P_s(\delta)$
- a good model for this is something like a [softmax](https://en.wikipedia.org/wiki/Softmax_function), [Boltzmann](https://en.wikipedia.org/wiki/Boltzmann_distribution), or [logistic function](https://en.wikipedia.org/wiki/Logistic_function)[^Boltzmann], i.e. a normalised ratio of exponentials, but this is not essential for the proof

The population alternates between a single parent and its two offspring, one of which is selected to become the next generation's parent.

- selection according to $P_s$ abstracts any mechanism which results in becoming the ancestor of future generations
  - not dying first
  - reproducing sufficiently successfully to 'win'
  - capturing/converting resources more efficiently
- in this first setup, all offspring are uniparental and no genes are mixed by any means other than mutation

![ Only one: an amoeba looks at a smaller amoeba with love and says, 'One day all this will be yours...'. In the next panel, the same amoeba looks at a second smaller amoeba, saying, '...unless it is yours. Depends who dies first.' ](https://raw.githubusercontent.com/olyis/sketch-repo/master/sketches/all-this-will-be-yours.jpg)

### Theorem

Now consider a binary fission from $\mathbf x_t$, yielding one perfect clone and one mutant clone $\mathbf x'_t$ where mutation $(\mathbf x_{t+1} - \mathbf x_t) \sim P_m$.

Define the next 'step', $\mathbf x_{t+1}$ as whatever mutant offspring $\mathbf x'_t$ eventually happens to be successfully selected vs a perfect clone, according to the selection function on their fitness differential. (If the perfect clone gets selected, it may take many intermediate generations to constitute a 'step' here.) Denote by $\alpha_s$ the resulting normalisation constant over mutations.

So the distribution over $\mathbf x_{t+1}$ given $\mathbf x_t$ is

$$P(\mathbf x_{t+1}|\mathbf x_t) = \alpha_s P_m(\mathbf x_{t+1} - \mathbf x_t)P_s(f(\mathbf x_{t+1}) - f(\mathbf x))$$

Call the mutation $\mathbf \epsilon = \mathbf x_{t+1} - \mathbf x_t$. Then we find

$$\begin{align*}
P(\mathbf \epsilon|\mathbf x_t) & = \alpha_s P_m(\mathbf \epsilon)P_s(f(\mathbf x_t + \mathbf \epsilon) - f(\mathbf x_t)) \\
  & \simeq \alpha_s P_m(\mathbf \epsilon)P_s(\mathbf \epsilon \cdot \nabla f(\mathbf x_t))
\end{align*}$$

by considering the directional derivative of $f$ along $\mathbf \epsilon$ at $\mathbf x_t$ and the limit as $\mathbf \epsilon \to \mathbf 0$[^directional]. (Prior to this infinitesimal limit, we have instead the _empirical approximation to_ the directional derivative.)

Now characterising $\mathbf \epsilon$ by length $r_{\mathbf \epsilon} = |\mathbf \epsilon|$ and angle-from-gradient $\theta_{\mathbf \epsilon} = \angle(\nabla f(\mathbf x_t), \mathbf \epsilon)$

$$P(\mathbf \epsilon|\mathbf x_t) \simeq \alpha_s g(r_{\mathbf \epsilon})P_s(r_{\mathbf \epsilon}|\nabla f(\mathbf x_t)|\cos \theta_{\mathbf \epsilon})$$

At this point it is clear that our step procedure depends, stochastically, on how closely the direction of the mutations match the fitness function's gradient.

By inspecting the expected value of the step direction, $\theta_{\mathbf \epsilon}$, we can make a more precise claim

$$\begin{align*}
\Bbb E[\theta_{\mathbf \epsilon}|\mathbf x_t] & = \int  P(\mathbf \epsilon|\mathbf x_t) \, \theta \, \mathrm d \mathbf \epsilon \\
  & \simeq \int \alpha_s g(r_{\mathbf \epsilon})P_s(r_{\mathbf \epsilon}|\nabla f(\mathbf x_t)|\cos \theta_{\mathbf \epsilon}) \, \theta_{\mathbf \epsilon} \, \mathrm d \mathbf \epsilon
\end{align*}$$

and finally, by noticing the integral of an odd function[^hyperspherical] in $\theta$

$$\Bbb E[\theta_{\mathbf \epsilon}|\mathbf x_t] \simeq 0$$

Thus the update between steps, $\mathbf \epsilon$, is a stochastic realisation of a variable whose orientation is, in expectation, exactly the same as that of the gradient $\nabla f(\mathbf x_t)$ of the fitness function.

By similar inspection of $\Bbb E[r_{\mathbf \epsilon}|\mathbf x_t]$ we can see that it is a monotonic function of $|\nabla f(\mathbf x_t)|$, depending on the particulars of $P_m$ and $P_s$, which together provide a gradient-dependent 'learning rate'.

So natural selection in this form really is nothing but an implementation of unbiased stochastic gradient descent!

## Discussion of simplifying assumptions

To what extent are the simplifying assumptions realistic? What happens to the equivalence when we relax any of the assumptions?

### Fixed 'fitness function'

In real natural selection, the interaction between a changing environment and a dynamic distribution of organisms collaborating and competing leads to a time- and location-dependent fitness function.

Variable fitness functions can lead to interesting situations like [evolutionarily stable](https://en.wikipedia.org/wiki/Evolutionarily_stable_strategy) equilibria with _mixtures_ of genotypes, or time- or space-cyclic fitness functions, or (locally) divergent fitness functions, among other phenomena.

Such a nonstationary fitness function is comparable to the use of techniques like self-play in RL, especially in conjunction with [Population-Based Training](https://www.deepmind.com/blog/population-based-training-of-neural-networks), but is less comparable to vanilla SGD.

As such it may be appropriate to think of real natural selection as performing something _locally_ equivalent to SGD but _globally_ more like self-play PBT.

### Continuous fixed-dimensional genome and radially-symmetric mutation probability density

Moving from a continuous to a discrete genome means that the notion of a gradient is no longer defined in the same way, but we can still talk about empirical approximate gradients and differences.

The mechanisms which introduce mutations in real natural selection are certainly symmetrical in certain ways, but probably not in any way which straightforwardly maps to radial symmetry in a fixed-dimensional vector space.

Without radial symmetry, much of the mathematics goes through similarly, but instead of an unbiased estimate of the gradient direction, it is biased by the mutation sampling. As such, we might think of real natural selection as performing a _biased_ stochastic gradient descent.

A comparison may be made to regularisation techniques (depending on whether they are construed as part of the training procedure or part of the objective function), or to the many techniques exploiting bias-variance tradeoffs in sampling-based gradient-estimation for RL, though these tend to be deliberately chosen with variance-reduction in mind, while natural selection may not exhibit such preferences.

### Limit case to infinitesimal mutation

In reality, mutations are not infinitesimal, but in practice very small relative to the genome. If we do not take the limit, instead of an exact directional derivative, we find an empirical-approximate directional derivative, yielding _empirical-approximate_ stochastic gradient descent.

This means that in real natural selection, the implied 'step size' or 'learning rate' is coupled with the particulars of the selection strength, the variance of the stochastic gradient, and the degree of empirical approximation applied. In contrast, stochastic gradient descent per se need not couple these factors together.

### A degenerate population of 1 or 2

If we expand the population to arbitrary size, it is possible to retrieve the equivalence with additional assumptions.

Instead of a parent _individual_ and cloned and mutated offspring _individuals_, considering parent and offspring _populations_, the same reasoning and proof is immediately applicable if we assume that _mutations arise sufficiently rarely_ to be either fixed or lost before the next mutation arises. In this case $P_s$, the probability of selection, becomes the probability of fixation.

Of course this is not true for real natural selection.

If instead we allow for multiple contemporary mutant populations, an identical treatment _can not_ be applied.

### No recombination or horizontal transfer

![ No crossing-over: a sad chromosome stands beside a sign depicting a crossed-out recombination event, with text 'No crossing-over' ](https://raw.githubusercontent.com/olyis/sketch-repo/master/sketches/no-crossing-over.jpg)

One of the most fascinating and _mathematically complicating_ aspects of real natural selection is _multiple heredity_ of genome elements, whether via horizontal transfer or sexual recombination.

The preceding proof of equivalence for natural selection and stochastic gradient descent rests on a model which does not include any notion of multiple heredity.

## Recovering the equivalence allowing arbitrary population size and recombination

Interestingly, the 'complicating' factor of multiple heredity provides a way to retrieve the equivalence in the presence of multiple contemporary mutations, as long as we continue to consider the limit of _infinitesimal mutations_.

For a single-heredity population, with multiple contemporary mutant subpopulations, we must either model 'only one winner', or model an ongoing mixture of subpopulations of varying sizes, either of which $P_s$ is unable to model without modification.

On the other hand, in a multiple-heredity population, assuming _eventually-universal mixing_, and crucially continuing to assume a _fixed fitness function_ (independent of the population mixture), a particular mutation must either fix or go extinct[^evo-stable].

### Proof sketch

So let us consider (instead of $\mathbf x_t$ and $\mathbf x_{t+1}$) $\mathbf x_{t_0}$ and $\mathbf x_{t_1}$, representing the _fixed part_ of the genotype at times $t_0$ and $t_1 > t_0$ respectively, that is the initial genome $\mathbf x_0$ plus all so-far-fixed mutations.

In the time between $t_0$ and $t_1$ the population will experience some integer number of mutation events (perhaps roughly Poisson-distributed but this is inessential for the proof), each of which is distributed according to $P_m$. Furthermore, at time $t_0$ some mutations from earlier times may be 'in flight' and not yet fixed or extinct.

Assuming fixed fitness, and infinitesimal mutations, we can represent the _probability of fixation by time_ $t_1$, namely $P_f$ with exactly the same properties as formerly assumed for $P_s$[^fixation]. Thus each mutation _fixed_ between $t_0$ and $t_1$ satisfies exactly the same unbiased-gradient-sampling property derived earlier, and so, therefore, does their sum $\mathbf x_{t_1} - \mathbf x_{t_0}$.

This relies on all in-flight mutations _not affecting_ the fitness differential, and thus $P_f$, of their contemporaries, which is certainly the case in the limit of infinitesimal mutations, but not the case for real natural selection.

### Summary of additional assumptions

7. Eventually-universal mixing

In particular, this means no speciation.

NB we also rely on 4. the limit to infinitesimal mutations, in an additional capacity. We also exclude all 'self-play-like' interactions arising from the larger population by relying further on 1. fixed 'fitness function'.

It may be feasible to retrieve a similar equivalence without excluding population-dependent fitness interactions with a different framing, for example considering gradients over 'mixed strategies' implied by population distributions.

## Conclusion

Natural selection, under certain conditions, carries out an implementation of stochastic gradient descent. As such, analogies drawn from one to the other are not baseless; we should, however, examine the necessary assumptions and be mindful of the impact of departures from those assumptions.

In particular, two sets of assumptions are presented here which together are sufficient to retrieve an equivalence:

1. Fixed 'fitness function' or objective function mapping genome to continuous 'fitness score'
2. Continuous fixed-dimensional genome
3. Radially symmetric mutation probability density
4. Limit case to infinitesimal mutation
5. A degenerate population of 1 or 2
6. No recombination or horizontal transfer

or, keeping assumptions 1 to 4 and relaxing assumptions 5 and 6

7. Eventually-universal mixing

This is not enough to cover all instances of real natural selection, but provides an approximate mapping from many instances.

Assumptions 2 and 3 together yield 'unbiased' SGD, and in their absence varying degrees of bias arise.

Assumption 1 rules out, most importantly, 'self play' and 'population-based' aspects of natural selection, which have other analogies in machine learning but which are firmly absent from vanilla SGD.

Further work could uncover other parameters of the emergent SGD, such as the variance of the implied gradient, the size of the implicit learning rate, the bias caused by relaxing assumption 3, or quantify the coupling between those factors.

Further scrutiny, especially of the assumptions related to population, 1, 5, 6, and 7, could better quantify the effect of making weaker or different assumptions.

[^directional]:

    The directional derivative in question is, for $\mathbf \epsilon = |\mathbf \epsilon| \mathbf u_{\mathbf \epsilon}$,
    
    $$\begin{align*}
    \lim_{|\mathbf \epsilon| \to 0} f(\mathbf x_t + \mathbf \epsilon) - f(\mathbf x_t) & = |\mathbf \epsilon| \lim_{|\mathbf \epsilon| \to 0} \frac {f(\mathbf x_t + |\mathbf \epsilon| \mathbf u_{\mathbf \epsilon}) - f(\mathbf x_t)} {|\mathbf \epsilon|} \\
      & = |\mathbf \epsilon| \left( \mathbf u_{\mathbf \epsilon} \cdot \nabla f(\mathbf x_t) \right) \\
      & = \mathbf \epsilon \cdot \nabla f(\mathbf x_t)
    \end{align*}$$

[^hyperspherical]:
    
    Cautious readers may note that the integral as presented is not posed in the right coordinate system for its integrand.
    
    By a coordinate transformation from Euclidean to [hyperspherical coordinates](https://en.wikipedia.org/wiki/N-sphere#Spherical_coordinates), centred on $\mathbf 0$, with $\nabla f(\mathbf x_t)$ providing the principal axis, $r$ the radial length, $\theta$ the principal angular coordinate, and $\mathbf \phi$ the other $n-2$ angular coordinates with axes chosen arbitrarily orthogonally,
    
    $$\begin{align*}
    \Bbb E[\theta|\mathbf x_t] & \simeq \int \alpha_s g(r)P_s(r|\nabla f(\mathbf x_t)|\cos \theta) \, \theta \, \mathrm d \mathbf \epsilon \\
      & = \int \alpha_s g(r) P_s(r|\nabla f(\mathbf x_t)| \cos \theta) \, \theta \, \left|\frac {\partial (r, \theta, \mathbf \phi)} {\partial (\mathbf \epsilon)}\right| \, \mathrm d r \, \mathrm d \theta \, \mathrm d \mathbf \phi \\
      & = \int_0^{\infty} \int_{-\pi}^{\pi} \alpha_s g(r) P_s(r | \nabla f(\mathbf x_t)| \cos \theta) \, \theta \left[ \int_{[0, \pi]^{n-2}} \left| \frac {\partial (r, \theta, \mathbf \phi)} {\partial (\mathbf \epsilon)} \right| \, \mathrm d \mathbf \phi \right] \, \mathrm d \theta \, \mathrm d r \\
      & = \int_0^{\infty} \int_{-\pi}^{\pi} \alpha_s g(r) P_s(r | \nabla f(\mathbf x_t)| \cos \theta) \, \theta \left[ \int_{[0, \pi]^{n-2}} r^{n-1} h(\mathbf \phi) \, \mathrm d \mathbf \phi \right] \, \mathrm d \theta \, \mathrm d r \\
      & = \int_0^{\infty} \int_{-\pi}^{\pi} \alpha_s g(r) P_s(r | \nabla f(\mathbf x_t)| \cos \theta) \, \theta \, kr^{n-1} \, \mathrm d \theta \, \mathrm d r \\
      & = \int_0^{\infty} \alpha_s k g(r) r^{n-1} \left[ \int_{-\pi}^{\pi} P_s(r|\nabla f(\mathbf x_t)| \cos \theta) \theta \, \mathrm d \theta \right] \, \mathrm d r \\
      & = 0
    \end{align*}$$
    
    where we use the fact that the hyperspherical Jacobian $\frac {\partial (r, \theta, \mathbf \phi)} {\partial (\mathbf \epsilon)}$ is independent of its principal angular coordinate $\theta$ and denote by $kr^{n-1}$ the result of integrating out the Jacobian over the other angular coordinates, and again noting that the symmetrical integral over an odd function is zero.

[^Boltzmann]:
    
    This can be justified in a few ways
    
    - If fitness is something like an [Elo rating](https://en.wikipedia.org/wiki/Elo_rating_system) then a Boltzmann distribution is implied
    - If we want to extend the two-individual case to the n-individual case but remain invariant to the arbitrary choice of 'baseline' fitness score, then a normalised ratio of exponentials is implied
    - We may further appeal to the maximum entropy property of Boltzmann distributions as a natural choice

[^evo-stable]:

    If we do not have a fixed fitness function, and in particular, if it is allowed to vary dependent on the distribution of the population, there are many [evolutionarily stable equilibria](https://en.wikipedia.org/wiki/Evolutionarily_stable_strategy) which can arise where some trait is stably _never_ fixed nor extinguished, but rather persists indefinitely in some proportion of the population. (A classic example is sex ratios.)
  
[^fixation]:
    
    We can be more precise if we have $P_f: \Bbb R^+ \to \Bbb R \to [0, 1]$ where the additional first parameter represents time-elapsed, so that $P_f(\delta t, \delta f)$ is the probability of a mutation with fitness delta $\delta f$ being fixed after elapsed time $\delta t$.
    
    Here we impose on $P_f(\delta t)$ (for fixed $\delta t$ time-elapsed) the same monotonicity requirement over fitness differential as imposed on $P_s$ before.
    
    The various 'in-flight' and intervening mutations in the proof also therefore implicitly carry with them $t_m$, the time they emerged, and the additional argument to $P_f$ is thus $\delta t = t_1 - t_m$.

    In practice we should expect $P_f$ to vary time-wise as a monotonically nondecreasing asymptote, but this property is not required for the proof.