Building Neural Networks in F# — Part 2
By: Haaris Mehmood
Tags: Machine Learning, Fsharp, Functional Programming, Neural Networks, Deep Learning

The Universal Approximation Theorem

First proven in 1989, neural networks can be classified as universal function approximators. But, what is such an approximator?

Given any continuous N-dimensional function f(x), a neural network with just a single hidden layer and a finite number of neurons has the ability to approximate such a function within a fixed range of x and a finite error ε.

We will make use of this theorem and evaluate a simple architecture on multiple non-trivial functions to prove that our network has the ability to learn from data.

Some Helper Functions

Let’s define these additional helper functions that will aid us in training and evaluation:

genDataSet1D — Using just the start, end and step information, this function generates an array of vector tuples with each tuple corresponding to an (x, y) pair. Note that this can be of any dimension as necessary but to keep things simple, both the inputs and outputs in this tutorial will be one-dimensional.

— Using just the start, end and step information, this function generates an array of vector tuples with each tuple corresponding to an pair. Note that this can be of any dimension as necessary but to keep things simple, both the inputs and outputs in this tutorial will be one-dimensional. yRealFn — This is where where we define our target function. Data points from this function will be sampled and plotted on the final graph to show the actual underlying function we are trying to model.

— This is where where we define our target function. Data points from this function will be sampled and plotted on the final graph to show the actual underlying function we are trying to model. yRealFnAndNoise — This function simply adds a probabilistic term to the output of yRealFn , simulating the presence of noise in data. If, however, this error is not scaled appropriately, it will be too large compared to the deterministic term (yRealFn), resulting in the network trying to learn noise rather than the actual function.

— This function simply adds a probabilistic term to the output of , simulating the presence of noise in data. If, however, this error is not scaled appropriately, it will be too large compared to the deterministic term (yRealFn), resulting in the network trying to learn noise rather than the actual function. meanFeatureSqErr & evalAngEpochErr —These functions will be explained in the training section of this tutorial.

Initializing the Network

Defining the Architecture

Sometimes, a single hidden layer is all we need…

For the purpose of testing our network, a simple neural network with just one hidden layer is sufficient. I played around with the number of hidden nodes necessary to learn any reasonable smooth function and found that 8 nodes were enough to do the job.

As a rule of thumb, choosing too few hidden nodes can lead to under-fitting and conversely too many can lead to over-fitting.

When it comes to choosing an activation function there are many popular ones at hand: tanh, sigmoid, ReLU, etc. I decided to go with tanh as its very well suited for learning non-linearities present in smooth functions such as those that will be used in our testing.

A plot of tanh(x) (red) and its derivative (green) [source].

Choosing the Hyper-parameters

If you have been following through machine learning, even from the very start you learn to appreciate that hyper-parameter search is more or less trial and error. Of-course, you can also look at cost function graphs over several epochs and make an educated guesstimate of what you think is the best value.

A learning rate of 0.03 with ~3000 epochs of training usually gave sufficiently good results. I was only able to work with these many epochs due to the surprisingly quick training times in F#!

The Target Functions

Before we can begin training, there is one last step: choosing reasonably complex functions the can be used to generate well curated data-sets and at the same time are comparable visually.

You would notice that outputs are bounded by the closed interval [0, 1] for all inputs that lie within the same range. This was a deliberate decision to avoid the need to use mean normalization which helps prevent over-blown weights by keeping all features within a similar range.

A definition of our three test functions along with their plots

We will train our network using x ∈ [0, 1] with a step-size of 0.01. Thus, 101 equally spaced samples in this range will be generated and we would reserve ~6% of these for validation (chosen uniformly).