## 6 Artificial Neural Networks  
ANNs are built from layers of neurons, like the Neural Network presented in Figure 1. Every single neuron in the network takes a weighted sum of all its inputs and returns, as output, the result of a non-linear activation function. The naming convention comes from biology, where the neuron is activated whenever the incoming signals from its synapsis excite it above a certain threshold. The output of one neuron is connected to all neurons in the next layer, where, again, a linear combination of all inputs is passed through a non-linear activation function.  
The process of ANN learning starts with random weight initialization for all neurons in all layers. The input vector is processed by the network and some output value is calculated. The output of the network will usually differ from the expected result (target), and so, the error of the network is calculated using a loss (error) function. As the next step, the error for each neuron is calculated and used by the optimization algorithm to change the weights of the model. This process is repeated for each sample (input vector) in the dataset. When the network has processed all vectors from the dataset, the process is repeated until the error satisfies some condition or a predefined number of repetitions is achieved. Every such iteration (processing of the entire dataset) is called an epoch in neural network training. The training can be therefore seen as the optimization process to find the minimum of the cost function, which is parametrized by the weights of the model. The weights, on the other hand, influence the decision function realized by the network itself. They change the way the network makes its predictions. If the error in the training set is very small, the network has learned how to exactly map the input vector to the output target value but may fail to generalize well to other inputs not seen during the training. Such a case is called overfitting, and we say that the model has high variance. The other extreme is when the model does not perform well on the training set, due to either not enough complexity or too few epochs in order to minimize the training loss (error). Such a model has high bias and the training results in underfitting. The correct training approach is to find a trade-off between variance and bias, choosing the model complexity that is enough to approximate the target function but not too complex to avoid overfitting, like in Figure 4. We need to include minimizing model complexity into the optimization target of the training function, as well as not create overly complex networks for trivial problems.  
The complexity of the model grows with the number of its trainable parameters (weights). Every non-zero weight increases complexity; therefore, keeping the absolute value of weights small, or even dropping some connections, contributes to fighting overfitting, hence producing simpler decision functions. This can be achieved by using a regularization term during training or adding special dropout layers to the network.  
If \(\mathcal{L}\) is a loss function, parametrized by the input vector \(X\), output value \(y\), and vector of all weights \(\theta\), then the regularized loss function, \(\hat{\mathcal{L}}\), will be increased by the Lp norm of \(\theta\). The regularization parameter \(\lambda\) specifies how much the norm of the weight vector influences the loss function and needs to be set as a hyperparameter before training:  
\[\hat{\mathcal{L}}(\theta,X,y)=\mathcal{L}(\theta,X,y)+\lambda\|\theta\|_{p}. \tag{6}\]  
It is visible that keeping the norm small minimizes the regularized loss function.