## 2 Neural Networks
### Working of a Neural Network  
The easiest way to understand how a neural network works is to note that each of the inputs (the \(x\) values) are provided as weighted inputs for the first hidden layer, and the outputs of the hidden layer become the inputs to each subsequent hidden layer. The weights vary for each node in every hidden layer and the output layer.  
In our example in Figure 1, the inputs \(x\) are passed to the first hidden layer \(y\), and at each node in the first hidden layer, the inputs are weighted to arrive at what is called the \"neuron input\". So for the network shown in Figure 1, there are two nodes in the hidden layer and the neuron input for each node is as follows, given the four input values \(x_{1},x_{2},x_{3},x_{4}\).  
\[a_{j}^{(1)}=\sum_{i=1}^{4}w_{ij}^{(1)}x_{i}+b_{j}^{(1)} \tag{3}\]  
where \(a_{j}^{(1)},j=1,2\) are the two neuron input values, and the weights are \(w_{ij}^{(1)},i=1,2,3,4;j=1,2\), which are the parameters of the model that need to be calibrated. The constant terms \(b_{j}^{(1)},j=1,2\) are the \"bias\" terms and cause a linear shift of the neuron input value. The superscripts denote the number of the hidden layer; here \"\({}^{(1)}\)\" denotes the first hidden layer. The first hidden layer has ten parameters (eight \(w\) values and the two bias \(b\) values). The second hidden layer has six parameters. Each node of the two hidden layers generates a single output through a nonlinear transformation of the neuron input, i.e., if we use sigmoid functions, then these would be as follows.  
\[y_{j}^{(1)}=\frac{1}{1+\exp(-a_{j}^{(1)})},\quad j=1,2 \tag{4}\]  
These values are then passed to the second hidden layer, with its own neurons that emit values into the final output layer, where the single sigmoid neuron produces a value between 0 and 1. This model example in Figure 1 has 19 parameters or weights, five for each node in the first hidden layer, three for each node in the second hidden layer, and three for the output node. In a typical deep neural network with several layers and many nodes per layer, the number of parameters can be much larger, hundreds of thousands to the low millions.  
The weights \(\{w_{ij}^{(r)},b_{j}^{(r)},w,b\}\), for all \(i,j\), and all hidden layers \(r\), are chosen to minimize the prediction error of the neural network on training data. The error is combined into a loss function across all or subsamples (known as batches) of the training data, usually by choosing the loss function to be the sum of squared prediction error, or some other loss function such as entropy. The calibrated weights will determine the accuracy of the network in predicting the output, i.e., the movement of the S&P index in our tests of market efficiency.  
The biggest challenge in calibrating an accurate neural network, is calculating the weights to be used at each layer. Today this challenge has been surmounted by special purpose hardware, and commoditized programming tools, such as Google's TensorFlow ([[https://www.tensorflow.org/](https://www.tensorflow.org/)]([https://www.tensorflow.org/](https://www.tensorflow.org/))) and other popular open source implementations of neural networks such as MXNet ([[http://mxnet.io/](http://mxnet.io/)]([http://mxnet.io/](http://mxnet.io/))), and h2o ([[https://www.h2o.ai/](https://www.h2o.ai/)]([https://www.h2o.ai/](https://www.h2o.ai/))). In conjunction with better hardware and software, one single mathematical innovation from more than three decades ago is solely responsible for the incredible efficacy of deep learning neural networks--backpropagation, often shortened to \"BackProp.\"