## 4 Seq2 seq Models  
The seq2 seq framework introduced by Google was initially applied to NMT tasks [13; 25]. Later, in the field of NLP, seq2 seq models were also used for text summarization [26], parsing [27], or generative chatbots (as presented in Section 2). These models can address the challenge of a variable input and output length. Formally, the seq2 seq task can be described as follows: let \((x_{1},x_{2},\ldots,x_{n})\) be an input (question) sequence and \((y_{1},y_{2},\ldots,y_{m})\) be an output (answer) sequence, not necessarily of the same length \((n\
eq m)\). The architecture of seq2 seq models contains two base components (see Figure 1): (1) anencoder, responsible for encoding the input \((x_{1},x_{2},\ldots,x_{n})\) into an intermediate representation \(h_{n}{}^{(Q)}\) (which is the last hidden state of the encoder), and (2) a decoder, responsible for decoding the intermediate representation \(h_{n}{}^{(Q)}=h_{0}{}^{(A)}\) into the output \((y_{1},y_{2},\ldots,y_{m})\). The conditional probability for \((x_{1},x_{2},\ldots,x_{n})\) to generate \((y_{1},y_{2},\ldots,y_{m})\) is presented in Equation (1). \[\mathbf{p}\Big{(}\mathbf{y}_{1},\mathbf{y}_{2},\ldots,\mathbf{y}_{m}\Big{|}x_{1},\mathbf{x}_{2}, \ldots,\mathbf{x}_{n}\Big{)}=\prod_{t=1}^{m}\mathbf{p}\Big{(}\mathbf{y}_{t}\Big{|}h_{n}^{ (Q)},\mathbf{y}_{1},\ldots,\mathbf{y}_{t-1}\Big{)}. \tag{1}\]  
Thus, the main goal of the task is to maximize the conditional probability \(max(p(y_{1},y_{2},\ldots,y_{m}|x_{1},\allowbreak x_{2},\ldots,x_{n}))\) that, for \((x_{1},x_{2},\ldots,x_{n})\), the most probable sequence \((y_{1},y_{2},\ldots,y_{m})\) would be found. More precisely, the encoder-decoder architecture is composed of several layers: (1) an encoder embedding layer, converting the input sequence into word embeddings: \((x_{1},x_{2},\ldots,x_{n})\rightarrow(x_{1}{}^{\prime},x_{2}{}^{\prime},\allowbreak \ldots,x_{n}{}^{\prime})\); (2) an encoder recurrent-based layer; (3) a decoder embedding layer, converting the output sequence into the word embeddings: \((y_{1},y_{2},\ldots,y_{m})\rightarrow(y_{1}{}^{\prime},y_{2}{}^{\prime},\ldots,y_{m}{}^{\prime})\); (4) a decoder recurrent-based layer; (5) a decoder output layer generating the conditional probability \(o_{t}=p(y_{t}|h_{n}{}^{(Q)},y_{1},\ldots,y_{t-1})\) for \(y_{t}\) from the hidden state \(h_{t}{}^{(A)}\) at each time step \(t\). Virtual words \(<\)bos\(>\) and \(<\)eos\(>\) represent the beginning and the end of the answer, respectively; the generation of the answer is terminated after generation of \(<\)eos\(>\). In the encoder and decoder layers, we use the recurrent-based approach (suitable for processing sequential data, i.e., text), but not the simple recurrent neural network (RNN), because it suffers from the vanishing gradient problem and, therefore, is not suitable for longer sequences. Instead of RNN, \begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline  & **Number of** & **Number of Tokens** & **Number of Tokens** & **Average \(Q\) Length** & **Average \(A\) Length** \\  & **Q and \(A\)** & **in Set \(Q\)** & **in Set \(A\)** & **in Tokens** & **in Tokens** \\ \cline{2-10}  & **EN/LT** & **EN** & **LT** & **EN** & **LT** & **EN** & **LT** \\ \hline Removed punctuation & 567 & 4025 & 2987 & 11,594 & 11,349 & 7.4 & 5.3 & 19.6 & 19.2 \\ \hline Separated punctuation & 567 & 4728 & 3549 & 14,818 & 14,531 & 8.5 & 6.8 & 25.4 & 24.9 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: English (EN) and Lithuanian (LT) datasets used in generative chatbots (Q and A abbreviations stand for questions and answers, respectively). we employ long short-term memory (LSTM) (introduced in Reference [28]) and bidirectional LSTM (introduced in Reference [29]) architectures, both having long memory. LSTM runs the input forward and, therefore, preserves information from the past, whereas bidirectional LSTM can run the input in two ways, forward and backward, thus preserving information both from the past and from the future in any hidden state.