All the single neurons
By: Cédric Bellet
Tags: Machine Learning, Neural Networks, Biffures

Here is my own representation of a single neuron — a bit more wavy, unpractical when it comes to drawing networks, hopefully not as bad for education purposes.

My own representation of a single neuron

A neuron has a body, dendrons to the left connecting to that body, synapses that are receptors for input coming to the neuron; to its right, the neuron has an axon, and an axon terminal (or multiple ones) that transfer the neuron’s information to the next one. The axon terminals connect to synapses.

Messages at the synapses are numbers, that are amplified or reduced as they pass through the dendrons, in proportion to the dendrons weights, noted w, or b in the case of the bias dendron (the bias dendron is always plugged to an incoming signal equal to 1). If a weight w is equal to 0, then the dendron is as good as dead — it passes no information at all. Signals from each dendron are summed in the neuron’s body, as the weighted input z.

The weighted input z then passes to the axon where it is optionally touched by an activation function σ. The activation function can be the Identity function (~the activation function does nothing to the weighted input), the Sigmoid function, the Rectified Linear function, but really, can be any function needed for a given application. What we get after the activation function is a¹, the neuron’s activation state.

Sample activation functions according to Wikipedia. Linear, non-linear, monotonous are not, symmetric or not… Activation functions can even be functions of not just the weighted input z, but of other parameters like weighted inputs in other neurons — see for example the softmax activation function.

From a place where I can actually use MathJax, here is what I have to say about the forward pass for our single neuron:

dot product is a scalar product, by opposition to the Hadamard product (or element-wise product) also used in machine learning

Putting single neurons to work

So far our neuron takes input and produces an output, but this is it. If the weights and bias are set in to certain values, the output can be useful — otherwise we just have a random number generator.

In order to make the neuron useful we need to train it. So we need a trainer for that neuron, that will (i) be able to evaluate the neuron’s performance, and (ii) create a learning procedure so the neuron performs better. We use a cost function to define (i), and the back-propagation algorithm to address (ii).

So what now? Well, let’s replace C and σ in the equations above with actual functions and we will start to see how those formulas are just generalizations of classic regression problems.

Single neurons solve (multi)linear regressions

The goal of linear regressions is to find, given a dependent variable and n explanatory variables, a line, expressed as a function of the explanatory variables such that the distance between the dependent variable and that line is minimal. A simple example of a linear regression is:

Here is exactly the same program, but written using the conventions we have used to describe our neuron so far:

in which we recognize a single neuron net problem, where the axon activation function σ is the identity function, and C is the squared l2 (or Euclidian) norm.

This neuron (Id activation function) with an squared l2 norm solves 3-parameter linear regressions

Single neurons solve logistic regressions

Similarly, logistic regressions can be expressed as single neuron network problems. Consider the stated goal of a logistic regression:

which is equivalent to the single neuron net program:

This neuron solves classification problems with 3 input parameters

In summary, using a single neuron, logistic vs linear regressions are simply a matter of flipping the activation and cost functions of the neuron:

A Python implementation and real regressions solved by a single neuron

Here is how the single neurons fared on four different regression exercises: