# Evaluation of the Coherence of Polish Texts Using Neural Network Models
## 2 Materials and Methods
### Coherence Estimation Models for The Polish Language  
#### 2.2.1 Semantic Similarity Graph  
Despite the effectiveness of the usage of neural networks to solve typical tasks, it is advisable to analyze the impact of different text features on the coherence evaluation process at first. Thus, it is suggested to consider the graph-based method of the coherence evaluation, namely, a semantic similarity graph [19]. Let us represent a text \(T\) as the set of sentences \(T=\{s_{1},s_{2},\ldots,s_{N}\}\) (\(N\) is a number of sentences); each sentence \(s_{i},i=1,2,\ldots,N\) is represented as the set of words \(\{w_{1}^{i},w_{2}^{i},\ldots,w_{m}^{i}\}\). The next step consists in the formalization of each sentence in a semantic space. It is necessary to perform sentence representation as a vector using a pre-trained semantic embedding model. In order to represent each word in a vector form, it is suggested to use ELMo model. Such a choice can be explained by the possibility of the architecture of this model to take into account the context of word usage while generating an output vector for it. The vector representation of each word is not unique and depends on surrounding words. Thus, each sentence \(s_{i}\) incorporates a set of vectors \(\big{\{}\mathbf{w}_{1}^{i},\mathbf{w}_{2}^{i},\ldots,\mathbf{w}_{m}^{i}\big{\}}\). Then, the averaged value of word vectors represents the sentence \(s_{i}\):  
Figure 3: The binary constituent tree that represents a sentence “[PERSON] was very hungry”. \[\mathbf{s}_{i}=\frac{1}{m}\sum_{k=1}^{m}\mathbf{w}_{k}^{i} \tag{2}\]  
According to the formed set of sentences, a semantic similarity graph \(G(V,E)\) is built where \(V\) is the set of vertices, \(E\) denotes the set of edges. Each vertex \(v_{i}\in V\) corresponds to a certain sentence, while a weighted directed edge \(e_{ij}\in E\) denotes the connection between sentences \(s_{i}\) and \(s_{j}\). There are three different approaches for the setting of edges \(e_{ij}\in E\): preceding adjacent vertex (PAV), single similar vertex (SSV), and multiple similar vertex (MSV). In the PAV approach, the connection is established just between adjacent sentences according to the order of sentences within a text. As a reader processes a text sentence by sentence, he or she can understand a current document's fragment basing on the previous text's parts. While designing a semantic similarity graph, such a cognitive process can be formalized using directed edges from the previous vertices (sentences) to the next ones. Firstly, an attempt to set an edge \(e_{ij}\) from the current vertex \(v_{i}\) to the previous one \(v_{j}\) is performed. If the value of the corresponding weight is non-zero, then the edge is set and the next vertex is chosen as current. In another case, the attempt to set an edge is performed for vertices \(v_{i}\) and \(v_{j-1}\). The weight of the edge \(e_{ij}\) denotes a semantic similarity between sentences \(s_{i}\) and \(s_{j}\); it is calculated by the following equation:  
\[\text{weight}(e_{ij})=\alpha\;\text{not}(s_{i},s_{j})+(1-\alpha)\cos(\mathbf{ s}_{i},\mathbf{s}_{j}), \tag{3}\]  
where not is the ratio of a number of common tokens of sentences \(s_{i}\) and \(s_{j}\) to a number of unique words; \(\cos(\mathbf{s}_{i},\mathbf{s}_{j})\) is a cosine distance between corresponding sentence vectors; \(\alpha\in[0,1]\) is a regularity parameter that allows taking into account the impact of different components on the accuracy of the method. In order to apply the not function, a lemmatization operation is performed for the words of the sentences \(s_{i}\) and \(s_{j}\). While building the graph \(G(V,E)\) with the usage of the SSV approach, it is possible to set edges between vertices regardless of their order. For each vertex \(v_{i}\), the search for the most similar vertex \(v_{j},i\
eq j\) is performed. The most similar vertex for \(v_{i}\) is chosen according to the highest weight of an edge \(e_{ij}\) among all possible connections. Then the corresponding edge \(e_{ij}\) is set.