You’ve probably missed one of the greatest ML packages out there
By: Oliver K. Ernst
Tags: Coding, Programming, Artificial Intelligence, Data Science, Machine Learning

Get Mathematica

Before you ask if it’s free — it is free-ish. Officially it’s proprietary and, of course, expensive — at the time of writing, a home desktop license runs $354. But there’s two large caveats here:

If you’re a student or affiliated with a university, your institution probably provides it for free — this is probably the most common exposure to this great tool.

If you’re not a university — Raspberry Pi OS (formerly Raspbian) ships with Mathematica for free. That means you can pick up a little $5 Raspberry Pi Zero (maybe even the ZeroW with built in WiFi!) and get a free working copy of Mathematica. From $354 down to $5 — how absurdly cool is that!

Three ways to make a neural network

Load a pre-made one = NetModel . Make a simple layer-to-layer one, like the ones you might see in an introductory ML textbook = NetChain . Make a general graph = NetGraph .

Let’s do each one in turn.

Load a pre-made network

The main command here is NetModel :

net = NetModel["LeNet Trained on MNIST Data"]

Note: You always can (and should) click the little plus button to get a more detailed breakdown of the network.

You can just fire up NetModel[] to see a list of all possible pre-loaded networks — I’ll put it at the bottom of this article.

Simple neural networks

A “simple” neural network in this case means a graph of layers where the i-th layer is connected to the i+1-st layer. This is done with the NetChain command:

net = NetChain[{

LinearLayer[10],

Tanh,

LinearLayer[20],

LogisticSigmoid

},

"Input" -> 10,

"Output" -> 20

]

This introduced a couple new things:

Layers: a NetChain is composed of layers that are connected together. A LinearLayer is just one like W * input + b . There’s lots more, like ElementwiseLayer , SoftmaxLayer , FunctionLayer , CompiledLayer , PlaceholderLayer , ThreadingLayer , RandomArrayLayer . Probably the most useful one is FunctionLayer , where you can apply a custom function (I’ll show you this layer).

is composed of layers that are connected together. A is just one like . There’s lots more, like , , , , , , . Probably the most useful one is , where you can apply a custom function (I’ll show you this layer). Activation functions: you can stick in pretty much any function you like! Tanh , Ramp (ReLU), ParametricRampLayer (leaky ReLU), and LogisticSigmoid are obvious choices, but you can use anything here. You can even repeat activation functions (for whatever reason) — no constraints here.

You can see the network is “uninitialized”. We can initialize the net with random weights and zero biases:

net = NetInitialize[net]

Let’s get those weights out. You can use Information[net, “Properties”] to query all the possible parameters. The layers can be extracted with:

layers = Information[net, "LayersList"]

linearLayers = Select[layers, ToString[Head[#]] == "LinearLayer" &]

The weights and biases can be obtained with:

weights1 = NetExtract[linearLayers[[1]], "Weights"]

Normal[weights1] // MatrixForm

biases1 = NetExtract[linearLayers[[1]], "Biases"]

Normal[biases1] // MatrixForm

Note that Normal converts NumericalArray to the standard Mathematica list object that we can manipulate easily.

Make any arbitrary graph

Mathematica is great at graphs. The main command to make neural networks is NetGraph . This is where it gets really cool!

Let’s start with a network that takes a target vector of size 2 and another vector of size 2, and computes the L2 loss:

layerDefns = <|



"l2" -> FunctionLayer[(#MyTarget - #MyLossInput)^2 &,

"MyTarget" -> 2, "MyLossInput" -> 2, "Output" -> 2],

"sum" -> SummationLayer[]

|>; layerConn = {{ NetPort["MyTarget"], NetPort["MyLossInput"]} ->

"l2" -> "sum" -> NetPort["MyLoss"]}; l2lossNet = NetGraph[layerDefns, layerConn]

We see the format for NetGraph :

An association (the Mathematica term for a dictionary of key/values) of name to layer.

(the Mathematica term for a dictionary of key/values) of name to layer. A list of layer names connected with arrows -> .

Also introduced is the notion of a NetPort . A NetPort is a named input/output of a network. You can clearly see the input ports on the left side of the layerConn definition, and the output ports on the right side:

layerConn = {{ NetPort["MyTarget"], NetPort["MyLossInput"]} ->

"l2" -> "sum" -> NetPort["MyLoss"]};

The inputs are "MyTarget" and “MyLossInput” , and the output is “MyLoss" . You can clearly see these in the graph as well:

Look at this carefully — it shows you the dimensions of each connection in the graph, which is extremely helpful. The inputs are vectors of length 2, while the output is a scalar.

You can now reuse this graph in yet larger graphs, for example with another subnetwork:

subNet = NetChain[{LinearLayer[2], Tanh, LinearLayer[10], Tanh,

LinearLayer[1]}]

and the combination of the sub network and the loss network:

layerDefns = <|

"sn1" -> subNet,

"sn2" -> subNet,

"l2loss" -> l2lossNet,

"catentate" -> CatenateLayer[]

|>; layerConn = {

NetPort["MyInput"] -> "sn1" -> "catentate",

NetPort["MyInput"] -> "sn2" -> "catentate",

{NetPort["MyTarget"], "catentate"} ->

"l2loss" -> NetPort["MyLoss"]

}; net = NetGraph[layerDefns, layerConn]

Let’s look at that more closely:

We have an input vector that get’s passed to in two separate channels of the subnetworks for a size of 2, then through different activation functions. These are then catenated (stacked), and passed into the loss function along with a target to finally obtain the loss.

Train networks

The command to train neural networks is NetTrain . Let’s train the net from the previous example. First, we need some data:

data = Flatten@Table[

<|

"MyInput" -> {x, y},

"MyTarget" -> {Cos[x + y], Sin[x + y]}

|>

, {x, -1, 1, .02}

, {y, -1, 1, .02}];

data[[1]]

We can see each data sample is of the format of an association (dictionary) with every NetPort labeled.

Next the training:

trainedNet = NetTrain[net, data, LossFunction -> "MyLoss"]

This is great — we get a simple overview and can visualize the training live. You can set different options — for example, the maximum number of iterations with:

NetTrain[..., MaxTrainingRounds->50000]

Finally, let’s evaluate the trained network. The network trainedNet outputs the full value of the loss function. Since we are interested more in the actual output, we can use NetTake to grab all layers up to and including the CatenateLayer :

trainedSubNet = NetTake[trainedNet, "catentate"]

trainedSubNet[data[[1]]]

data[[1]]["MyTarget"]

Let’s plot that and do some regression!

pts = Flatten[Table[

{x, y, trainedSubNet[{x, y}][[2]]}

, {x, -1, 2, .05}

, {y, -1, 2, .05}], 1];

ptsTrain = Select[pts, #[[1]] <= 1 && #[[2]] <= 1 &];

ptsExtrapolate = Select[pts, #[[1]] > 1 || #[[2]] > 1 &];

pltPtsTrain = ListPointPlot3D[ptsTrain, PlotStyle -> Black];

pltPtsExtrapolate = ListPointPlot3D[ptsExtrapolate, PlotStyle -> Blue];

pltTrue =

Plot3D[Sin[x + y], {x, -1, 2}, {y, -1, 2},

PlotStyle -> Opacity[0.3], Mesh -> None, PlotLabel -> "Sin[x+y]",

BaseStyle -> Directive[FontSize -> 24]];

Row[{

pltTrue,

Show[pltPtsTrain, pltPtsExtrapolate, PlotRange -> All],

Show[pltPtsTrain, pltPtsExtrapolate, pltTrue, PlotRange -> All]

}]

Final thoughts

You can read Mathematica’s more complete guide on neural networks here. However, it’s bit long…. If you prefer to just read through the API pages, here’s a better list from Wolfram.