## 3 Deep Learning Methods for Text Processing
### Feedforward Neural Network  
Deep forward neural networks known as the feedforward neural networks (FFNN) or multilayer perceptrons are basic models of deep learning. Feedforward networks became popular in 1986 when [PERSON], [PERSON], and [PERSON] introduced a method of training forward neural networks using the error back-propagation [37]. The goal of feedforward neural networks is to approximate the function \(f^{*}\). For example, the function \(y=f^{*}(x)\) maps the input \(x\) to the value \(y\). FFNN defines the mapping \(y=f(x;\theta)\) and finds the value of the parameters \(\theta\), which leads to the best approximation of the function. The flow of information in FFNN is forward; in practice, this means that the computational model represents an acyclic graph. The basic model of a neuron is called perceptron. The perceptron receives input signals \(\bar{x}=(x_{1},x_{2},\dots,x_{n+1})\) via synaptic weights, which form the vector \(\bar{w}=(w_{1},w_{2},\dots,w_{n+1})\). The perceptron output is given as the scalar product of the input vectors transformed using the activation function \(f\), to which the bias is added. Bias \(b\) is a constant that does not depend on the input parameters and serves to influence the activation function [38; 39]. \[output=f(\bar{w}\cdot\bar{x})=f\bigg{(}\sum_{i=1}^{n+1}w_{i}x_{i}\bigg{)}+b. \tag{1}\]  
For the best classifiers in our work are used following hyper-parameters and settings:  
* Activation function on input and hidden layers: ReLU: In proposed solution we used ReLU (Rectified Linear Unit) [40] activation function. ReLU belongs to one of the most frequently used activation functions applied in deep networks. It is defined as: \[f(x)=\max(0,x)\] (2)which means, that it transforms negative inputs to 0 and leaves positive inputs without transformation. * Loss function: binary cross entropy Error function is used to estimate the error of the model during the training. Using the error backpropagation [41], the neuron weights in particular layer are updated in such manner, that the error rate decreases in following evaluation. We used Binary Cross-Entropy (BCE), which can be defined as: \[\text{BCE}=-\left(y\log\left(\hat{y}\right)\left(1-y\right)\log\left(1-\hat{y} \right)\right),\] (3) where \(y\) represents the ground truth and \(\hat{y}\) represents predicted value. * Optimization: Adam To minimize the error rate of the model in the prediction, optimization function is used. We used Adam (Adaptive Moment Estimation) [42]. Adam is an optimization function, which computes the learning estimation for each parameter. In addition to storing an exponentially decaying average of past squared gradients \(v(t)\) like RMSprop [43], Adam also keeps an exponentially decaying average of past gradients \(m(t)\), similar to momentum [44]. Both moving averages are initialized to 0, which leads to the moments' estimation biased towards zero. Such situation occurs mostly during the initial phases when decay parameters (\(\beta 1\), \(\beta 2\)) have values close to 1. Such biased can be removed using modified estimations \(\hat{m}_{t}\) a \(\hat{v}_{t}\): \[\hat{m}_{t}=\frac{m_{t}}{1-\beta_{1}^{t}}\qquad\hat{v}_{t}=\frac{v_{t}}{1- \beta_{2}^{t}},\] (4) The parameters are updated according to the formula: \[\theta_{t+1}=\theta_{t}\frac{\alpha}{\sqrt{\hat{v}_{t}}+\epsilon}\cdot\hat{m} _{t}.\] (5) Default setting for the parameter \(\beta_{1}\) is 0.9 and \(\beta_{2}=0.999\), learning rate \(\alpha=0.001\) and \(10^{-8}\) for the \(\epsilon\) parameter. Adam is considered as a suitable optimization method in practical tasks. When comparing Adam to other methods, its advantage is faster convergence and training speed is also higher. It also removes certain issues of other optimization techniques, such as slow convergence, or high variance of the parameters, which could lead to variations in the error function.