# Evaluation of the Coherence of Polish Texts Using Neural Network Models
## 3 Experiments and Results
### Results  
After the preparation of the considered models, the accuracy of the semantic similarity graph (SSG), the LSTM-based model (LSTM), and the BERT-based model (BERT) was calculated according to the solving of the sentence ordering and the insertion task. The results obtained are shown in Table 1.  
Firstly, let us analyze the obtained results for the different approaches of the semantic similarity graph. In the case of the PAV approach, the increase of both metrics is tracked during the increase of the regululative parameter \(\alpha\) till the reach of the peak with \(\alpha=0.6\)  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model** & **Parameter Value** & **Sentence Ordering** & **Insertion Task** \\ \hline _SSG (PAV), parameter \(\alpha\)_ & 0.0 & 0.786 & 0.105 \\  & 0.1 & 0.788 & 0.109 \\  & 0.2 & 0.794 & 0.107 \\  & 0.3 & 0.792 & 0.131 \\  & 0.4 & 0.790 & 0.133 \\  & 0.5 & 0.796 & 0.131 \\  & 0.6 & 0.807 & 0.138 \\  & 0.7 & 0.799 & 0.131 \\  & 0.8 & 0.786 & 0.116 \\  & 0.9 & 0.757 & 0.112 \\  & 1.0 & 0.740 & 0.098 \\ \hline _SSG (SSV)_ & - & 0.760 & 0.083 \\ \hline _SSG (MSV), parameter \(\theta\)_ & 0.0 & 0.759 & 0.107 \\  & 0.1 & 0.718 & 0.092 \\  & 0.2 & 0.696 & 0.077 \\  & 0.3 & 0.716 & 0.072 \\  & 0.4 & 0.659 & 0.105 \\  & 0.5 & 0.731 & 0.105 \\  & 0.6 & 0.665 & 0.094 \\  & 0.7 & 0.584 & 0.077 \\  & 0.8 & 0.525 & 0.055 \\  & 0.9 & 0.055 & 0.002 \\ \hline _LSTM_ & - & 0.875 & 0.168 \\ \hline _BERT_ & - & 0.835 & 0.143 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Accuracies of the solving of different tasks for coherence estimation models on a Polish corpus.  
The further increase of the \(\alpha\) parameter decreases corresponding metrics. Such a value \(\alpha=0.6\) may indicate the necessity to take into account both lexical and semantic components of a text simultaneously while evaluating the coherence of a document. Moreover, the highest metric values for the semantic similarity graph method were retrieved using the PAV approach that underlines the expediency of the tracking of the connection between sentences sequentially. Such a conclusion can be drawn from the metrics of the SSV and MSV approaches, too. Despite the change of the regulative parameter \(\theta\), all corresponding metrics of the MSV approach are lower than the corresponding values of the SSV approach. Thus, it is advisable to analyze a text in a sentence-by-sentence manner according to their positions while estimating the coherence of a Polish corpus.  
The metrics for other models based on neural networks outperform corresponding values of the semantic similarity graph approach. The highest values of the LSTM-based model for both tasks may indicate the possibility to apply different recurrent neural networks in order to estimate the coherence of Polish texts. Moreover, the word order can be taken into account during the coherence evaluation of Polish documents despite the relevance of the Polish language to the category of synthetic languages. As for the BERT-based model, the obtained metrics may indicate the ability of such architecture to resolve typical tasks even with the usage of externally pre-trained parameters. The increase of the corresponding values for this approach can be achieved by the fine-tuning of all parameters of the BERT model according to the coherence estimation task.