# Evaluation of the Coherence of Polish Texts Using Neural Network Models
## 2 Materials and Methods
### Coherence Estimation Models for The Polish Language
#### 2.2.3 Using of the BERT Model  
While considering the representation of multi-language texts, it is advisable to pay attention to the Transformer architecture [23]. This architecture is widely used in different NLP tasks, especially, for machine translation purposes. The key feature of the Transformer-based approach consists of an attention mechanism that implies the revealing of connections between different words of a sentence. Figure 6 demonstrates the example of such attention from each left token to another right. The darker color of lines indicates higher attention weight between tokens. Tokens [CLS] and [SEP] denote the start and the end of sentences correspondingly. The sets of such attention mechanisms (so-called \"heads\") allow the effective encoding of input sequence with further decoding of it for the representation of translated sequence without the loss of information about word connections. The availability of such connection between tokens, regardless of word order, can be important for synthetic languages like the Polish language, where the order of words is not so valuable in comparison with analytical languages. Thus, it is suggested to consider the impact of such attention mechanisms on the coherence evaluation process of Polish texts. The classical Transformer model incorporates two consecutive parts: an encoder and a decoder. According to the coherence estimation process, it is necessary to perform the vector representation of input data for further binary classification. Thus, the encoder part of a Transformer-based model should be used. In order to perform such representation, the Bidirectional Encoder Representations from Transformers(BERT) model can be utilized. The training of this Transformer-based model provides for bidirectional signal flow. It means that the processing of a text incorporates both left-to-right and right-to-left signal directions. The results obtained showed that such an approach allows better recognition of context in comparison with a single-direction mode. In the case of the analysis of synthetic languages, it can help to reveal more dependencies between tokens regardless of word order. As the BERT model is designed to generate a language model, only the encoder part of the Transformer architecture is trained. Thus, the pre-trained BERT model can be utilized during the coherence evaluation process of a Polish-language corpus. Let us formulate the coherence estimation task according to the previously considered LSTM-based model. Thus, the representation of an input text and its coherence calculation are described by Equations (5) and (6). The key task consists in the prediction of the coherence of an input clique (a group of sentences). Figure 7 shows the main components of the coherence estimation model based on the pre-trained BERT layers. Firstly, input sentences are passed through a preprocessing block. In contrast to the previously considered model, all sentences are processed as a whole block. The preprocessing implies the tokenization of the input text block using a pre-defined BERT tokenizer. Thus, the output of the block is represented as a set of numbers \(\{t_{1}^{1},t_{2}^{1},\dots\}\) where each number corresponds to an input token according to the vocabulary \(V\) of the tokenizer. Moreover, as the Polish language has different grammatical cases, the closed version of both the tokenizer and the BERT model are suggested to be utilized. The next component is the pre-trained BERT model itself. In order to verify how the representation of the input text by the pre-trained BERT model can fit coherence requirements, it is proposed to \"freeze\" all parameters of this component during a training process. Such an approach can help verify whether this neural network architecture is suitable for the coherence evaluation task at all. The BERT model incorporates the following parts:  
* Embedding that processes the sequences of input token numbers and attention masks in order to perform the dense representation.