## 5 Methods
### Artificial Neural Networks
The weight space of the neurons is moved against the direction of the gradient of \(E_{\Delta}\) with respect to each weight  
\[\frac{\partial E_{\Delta}}{\partial w_{ij}}=\frac{\partial\frac{1}{2}\big{(}t _{j}-o_{j}\big{)}^{2}}{\partial w_{ij}}. \tag{27}\]  
As we differentiate with respect to the weights of \(j\)th neuron, the weights of the other neurons in the sum vanishes. Inserting  
\[o_{j}=\phi\bigg{(}\sum_{i}w_{ij}x_{i}\bigg{)}=\phi(v) \tag{28}\]  
and applying the chain rule results in  
\[\frac{\partial E_{\Delta}}{\partial w_{ij}}=-\big{(}t_{j}-o_{j}\big{)}\cdot x _{i}\cdot\phi^{\prime}(v). \tag{29}\]  
Once again, a learning rate is used for weight update and the minus sign is eliminated for moving in the direction of the minimum. This results in the updated equation  
\[\Delta w_{ij}^{delta}=\alpha\big{(}t_{j}-o_{j}\big{)}\cdot x_{i}\cdot\phi^{ \prime}(v). \tag{30}\]  
The delta rule is only applicable to single layer-layer neural networks. A generalized form is the Error Backpropagation learning. It uses the very same error function to minimize:  
\[E_{ebp}=\sum_{j}\frac{1}{2}\big{(}t_{j}-o_{j}\big{)}^{2}. \tag{31}\]  
However, the adjustment of the weights depends on whether the neuron is in the output layer or in the hidden layer:  
\[\Delta w_{ij}^{ebp}=-\alpha\frac{\partial E}{\partial w_{ij}}=\alpha\delta_{ j}o_{i} \tag{32}\]  
and  
\[\delta_{j}= \begin{cases}\phi^{\prime}(v)\big{(}t_{j}-o_{j}\big{)}&\text{if $j$ corresponds to an output neuron}\\ \phi^{\prime}(v)\sum_{k}\delta_{k}w_{jk}&\text{if $j$ corresponds to hidden neuron}\end{cases} . \tag{33}\]Here, \(i\) corresponds to prior neurons of \(j\) and \(k\) corresponds to subsequent neurons. This means that \(o_{i}\) is the input of neuron \(j\), for the first layer \(o_{i}\) equals \(x_{i}\). Moreover, for hidden neurons the weight adjustment depends on the error of the subsequent neurons. Thus, the error is back propagated to hidden neurons. The Error Backpropagation algorithm is also a gradient decent based, which makes it vulnerable for finding local minima. Therefore, different solutions are possible for different weight initialization. Secondly, the algorithm tends to oscillate when the learning rate is too large. Several approaches make use of a momentum term in order to smooth the solution process. The tendency for oscillations can also be tackled by lowering the learning rate constant. However, this also decreases the training speed. In contrast, the directional search extension can speed up the solution process. The idea is to look for a minimum in gradient direction before the calculation for a new gradient. The minimum in gradient direction can be found using the Newton-Raphson method. In Recurrent Neural Networks, the Error Backpropagation algorithm cannot be used in its original form due to the recurrent loops preventing it. With the help of the Backpropagation Through Time method, this problem can be overcome. Backpropagation Through Time is a method based on Backpropagation. Here, the Error Backpropagation method is extended by a time factor. As the output of a Recurrent Neural Network depends on the order of the data, the training data are an ordered sequence of \(k\) input and output pairs \(\langle x_{1},y_{1}\rangle,\langle x_{2},y_{2}\rangle,\ldots,\langle x_{k},y_ {k}\rangle\). In order to subsequently apply the original Backpropagation algorithm, the Recurrent Neural Network must be unfolded. The unfolded network thus contains \(k\) inputs. The recurrent components of the original network are completely duplicated for each required time step. The Backpropagation algorithm is then applied to the expanded network. The complexity of the resulting unfolded network increases drastically when increasing the length of the training sequence. This can lead to problems when using the Backpropagation algorithm as, as shown in the corresponding Equation (33), the variations of the weights also depend on the gradients of the errors of the subsequent neurons.