# Financial Context News Sentiment Analysis for the Lithuanian Language
## 4 Experimental Investigation  
To perform the experimental investigation to find the best classifier for the analyzed dataset, cross-validation and hyperparameters optimization with a grid search optimizer was used. Each experiment was repeated five times and the average accuracy was evaluated. For cross-validation, the commonly used 5-fold option was selected (80% training dataset, 20% testing dataset); moreover, the so-called stratify distribution was used, which helped to keep the same number of the class in each fold. Many parameters can influence the LSTM classifier results, but the main are sequence length (tokens number in the text), embedding dimension, hidden unit number, epoch, and of course, the LSTM architecture. In our research, we used the simple LSTM architecture because the analyzed dataset was quite small: sequence layer, word embedding layer, LSTM layer, dropout layer, fully connected layer (three class), softmax layer, and classifier layer. As we can see, one of the most complicated long short-term memory algorithms has many, various options; thus, first of all, the primary experiments were made to find the best epoch number for training. For this reason, all datasets were divided into three-part subsets: 80% training, 10% validation, and 10% testing. The related work review showed that, for LSTM models, the validation dataset was needed because it helped check the training process and avoid the underfit or overfit model problems. Two parameters of the LSTM algorithm were fixed regarding the primary experiments: dimension was equal to 50 and the epoch number was equal to 10. As we can see in Figure 5, the validation (black line) and training (orange line) loss curves became stable around epoch 2, meaning that further training does not make sense, the model does not learn. The hyperparameters optimization helps to find the best parameters of each classifier, which allows obtaining the highest model accuracy. The gird search was performed for two algorithms: a long short-term memory and a support vector machine. The multinomial Naive Bayes did not have a parameter to optimize, so for this model, the grid search was not used. All parameters, their bound ranges, obtained results, and all other important details of the experimental investigation, are presented in Table 3.  
Figure 5: The LSTM training process of the analyzed dataset: training loss (orange), validation loss (black). \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Classification Algorithm** & \multicolumn{1}{c}{**Dataset Preprocessing**} & \multicolumn{1}{c}{**Grid Search**} & \multicolumn{1}{c}{**Best Parameters**} & \multicolumn{1}{c}{**Average Accuracy**} \\ \cline{3-5}  & **Balance** & **Parameter/Bounds** & **\%** \\ \hline \multirow{6}{*}{Naive Bayes} & \multirow{2}{*}{Non-preprocessing} & POS balance & 70.8 \\ \cline{3-5}  & & NEU balance & 62.9 \\ \cline{3-5}  & & NICE balance & 65.2 \\ \cline{3-5}  & & POS balance & 71.1 \\ \cline{3-5}  & \multirow{2}{*}{Proprocessed} & NEU balance & 63.3 \\ \cline{3-5}  & & NICE balance & 64.4 \\ \hline \multirow{6}{*}{Superpert vector machine} & \multirow{2}{*}{Non-preprocessing} & POS balance & RX: 0.4646; 35 \(<\) : RX: linear; \(S\): false. & 67.4 \\ \cline{3-5}  & & NEU balance & Box constraint (RX): value log-scaled in the range [k]: 58 \(<\) : RX: linear; \(S\): false. & 59.8 \\ \cline{3-5}  & & NICE balance & [k-3]: 16.0 \\ \cline{3-5}  & & POS balance & Kernel scale (SS) value log-scaled in the range [k-3]: 16.5 & RX: 0.0215; \(S\):  
The comparison results, using a non-preprocessed and preprocessed dataset, with three balancing methods, are presented in Figure 6. As we can see, the accuracy results of the multinomial Naive Bayes algorithm almost, in all cases, were slightly higher, just in the cases of using the preprocessed dataset and NEU balance, the accuracy was the same as an LSTM algorithm (63.3%). Moreover, there was no significant difference between non-preprocessed and preprocessed dataset results.