# Stock Market Crisis Forecasting Using Neural Networks with Input Factor Selection
## 4 Model Building with Neural Networks
### Structure of the Neural Networks
Then, the activation function \(a_{1}(\cdot)\) at this layer, for which we take the _rectified linear unit function (ReLU)_, processes \(P^{1}_{k_{1},t}\) and returns the output of the neuron \(k_{1}\): \[P^{1,+}_{k_{1},t}=a_{1}(P^{1}_{k_{1},t}+c^{1}_{k_{1}})=max\{P^{1}_{k_{1},t}+c^ {1}_{k_{1}},0\}\,,\] (9) where \(c^{1}_{k_{1}}\) denotes the _bias_ at the neuron \(k_{1}\). 2. At the output layer (\(h=2\)), the outputs from the neurons of the previous hidden layer are treated as the inputs for the respective propagation functions. Therefore: 1. At the output neuron, the propagation function has the form: \[P^{2}_{t}=\sum_{k_{1}=1}^{3}w^{2}_{k_{1}}P^{1,+}_{k_{1},t}\,,\] (10) where \(w^{2}_{k_{1}}\) is the weight for the \(k_{1}\)-th neuron of the hidden layer. 2. Then, the _sigmoid activation function_\(a_{2}(\cdot)\) at this neuron processes \(P^{2}_{t}\) and returns the final output of this neuron: \[\hat{y}_{t}=P^{2,+}_{t}=a_{2}(P^{2}_{t}+c^{2})=\frac{1}{1+e^{-(P^{2}_{t}+c^{2} )}}.\] (11)  
Figure 2: Feed-forward neural network with one hidden layer. After obtaining the final output at a time \(t\)\((t=1,\ldots,T)\), we use a _binary cross-entropy loss function_ to measure the _losses_ between the outputs from the neural network, \((\hat{y}_{t})_{t=1,\ldots,T}\), and the real observation of the state-identifier, \((y_{t})_{t=1,\ldots,T}\). Since there are far more non-crisis dates than crisis dates, our classification problem is not well-balanced and there is the risk that the crisis dates are not sufficiently taken into account during the fitting process. This is crucial as a crisis forecasting system should work in a reliable way, in particular in times of crises. To overcome this problem, we use a _weighted binary cross-entropy loss function_ (see, e.g., [24]), defined as:  
\[\ell=-\sum_{t=1}^{T}[c_{1}y_{t}\log(\hat{y}_{t})+c_{0}(1-y_{t})\log(1-\hat{y}_{ t})],\]  
where \(y_{t}\) is either \(0\) or \(1\), indicating which state the real observation is in, and the output \(\hat{y}_{t}\) from the neural network can be interpreted as the probability that this instance is predicted to be in state \(1\). \(c_{0}\) and \(c_{1}\) are the weights. Setting a higher value for the weight of the class to which less data points belong punishes missed classifications in this class. In our case, we have less crisis dates (class \(1\)) than non-crisis dates (class \(0\)). In order to prevent the non-crisis dates dominating the training, we set a higher weight for \(c_{1}\) than for \(c_{0}\). We choose these weights to be inversely proportional to the number of data points of the corresponding class in the data set (see, e.g., [25]). We utilize the commonly used _back-propagation_ to minimize the losses from the weighted binary cross-entropy loss function. Back-propagation computes the gradient of the loss function at each neuron with respect to its weights and biases, and then updates its weights and biases by subtracting the product of the gradient and the learning rate \(\gamma\) from them (see, e.g., [23]).