## 4 Seq2 seq Models
Both encoder and decoder are jointly trained; errors are backpropagated through the entire model and weights are adjusted. The encoder-decoder weights are adjusted in 200 epochs with the batch size equal to 5. Three different seq2 seq architectures (in Figures 2-4) explored in our experiments were implemented with the functional API using Tensorflow [30] (the python library used for dataflow and machine learning) with Keras [31] (the python library for deep learning). Used architectures were plotted with the _plot_model_ function in Keras. Figure 1: The basic recurrent-based encoder-decoder architecture. Figure 2: One-layer long short-term memory (LSTM) for encoder and decoder (abbreviated as LSTM encoder and decoder). E1, E2 represent the encoder; D1, D2, D3 represent the decoder. All seq2 seq architectures were applied on top of word embeddings. In our experiments, we used the following types:  
* One-hot encoding. This is a discrete representation of words mapped into corresponding vectors containing all zero values, except for one value equal to 1. Thus, each word is represented by its unique vector with the value of 1 at a different position. A length of such a vector becomes the vocabulary size. Sizes of question and answer vocabularies are not necessary equal; therefore, the lengths of their one-hot encoding vectors are also different. * FastText (presented in Reference [32]). FastText is a library (offered by Facebook's AI Research Lab) used to train neural distributional embeddings able to catch semantic similarities between words. In this research, we experiment with English (cc.en.300.vec) and Lithuanian (cc.lt.300.vec) fastText embeddings trained using continuous-bag-of-words (CBOW) with position-weights, in 300 dimensions, with character 5-g of window size five and 10 negatives [33]. Each fastText word embedding is a sum of \(n\)-gram vectors incoming into that word (e.g., 5-g _clath_, _hatbo_, _atbot_\(n\)-grams compose the word _clathbot_). FastText word embeddings can be created even for misspelled words,  
Figure 4: One-layer bidirectional LSTM (BiLSTM) encoder and one-layer LSTM decoder (abbreviated as BiLSTM encoder and LSTM decoder). E1, E2, E3, E4_h, E4_c represent the encoder; D1, D2, D3 represent the decoder. Figure 3: Two-layer stacked LSTM encoder and one-layer LSTM decoder (abbreviated as LSTM2 encoder and LSTM decoder). E1, E2, E3 represent the encoder; D1, D2, D3 represent the decoder. and the obtained vectors are close to their correct equivalents. This is an advantage for languages having the missing diacritics problem in non-normative texts. Despite the Lithuanian language having this problem, the dataset that we used in this research contained only normative texts. * BERT (Bidirectional Encoder Representations from Transformers) (offered in Reference [34]). This is Google's neural-based technique (with multi-directional language modeling and attention mechanisms), which demonstrates state-of-the-art performance on a wide range of NLP tasks, including chatbot technology. Distributional BERT embeddings are also robust to disambiguation problems, i.e., homonyms with BERT are represented by different vectors. In our experiments, we used the BERT service [35] with the base multilingual cased 12-layer, 768-hidden, 12-head model for 104 languages (covering English and Lithuanian). Different vectorization types were tested in the following encoder-decoder units:  
1. One-hot encoding with both encoder and decoder;
2. FastText (abbreviated ft) embeddings with the encoder and one-hot encoding with the decoder;
3. BERT embeddings with the encoder and one-hot encoding with the decoder;
4. FastText embeddings with both encoder and decoder;
5. BERT embeddings with both encoder and decoder.