## 8 Experiments
### Model Settings  
Experiments were performed using three different types of modeling units: words, subwords, and characters. All words that occur in the training sets were added to the associated vocabularies of words. The statistics are presented in Table 5. Subwords were obtained with the tool SentencePiece [33] and the unigram-based splitting. Experiments were performed with three different vocabulary sizes: 5000, 1000, and 500. In experiments with characters as the basic units, using character sequences for segments would be too lengthy for the tools, so we used overlapping word 3-grams instead for training and inference. Overlapping word 3-grams ensure the left and right context of one word. In all other experiments, segments were applied. \begin{table}
\begin{tabular}{l r r r r} \hline
**Dataset** & **Training** & **Development** & **Evaluation** & **Total** \\ \hline GosVL & 9056 & 1132 & 1132 & 11,320 \\ Gos & 88,973 & 11,122 & 11,122 & 111,217 \\ [PERSON]_J & 45,515 & 5689 & 5689 & 56,893 \\ [PERSON]_N & 39,494 & 4937 & 4937 & 49,368 \\ [PERSON]_P & 62,292 & 7787 & 7787 & 77,866 \\ All & 245,327 & 30,666 & 30,666 & 306,659 \\ \hline \end{tabular}
\end{table} 
Table 4: Number of segments in each dataset that are used for training, development, and evaluation. In the first set of experiments, statistical translation models were built using the Moses SMT toolkit [51] and SRILM toolkit [52] for training language models. Character-based translation models were investigated first, followed by word-based and subword-based models. For character-based translation models, a 10-g language model was applied; for word-based models, a 3-g language model; and for subword models, a 5-g language model. In the second set of experiments, neural network-based translation models were built using the Marian NMT toolkit [53]. LSTM and transformer architecture were used. The most important hyperparameters used in experiments are collected in Table 6. Other hyperparameters were left at their default values from the respective tools. We used the SacreBLEU [46] tool for computing BLEU and chrF scores. For a deep inspection of the results and to perform statistical tests, we developed our own tool. Statistical models were trained on Intel Xeon E5-2699 v4 CPUs, while LSTM and transformer models were trained on nVidia Tesla V100 GPUs. For the largest training set (All), the training times for statistical models ranged from 12 min (for word-based models) to 80 min (for character-based models). The training times for LSTM and transformer \begin{table}
\begin{tabular}{l r r} \hline
**Approach** & **Parameter** & **Values** \\ \hline SMT, SRILM & alignment tool & GIZA++ \\  & language model order & 3, 5, 10 \\  & language model smoothing & Good-Turing discounting \\  & gtnmin & 1 \\  & alignment & grow-diag-final-and \\  & distortion limit & 0 \\  & max phrase length & 10 \\ \hline LSTM & valid-metrics & ce-mean-words \\  & valid-mini-batch & 64 \\  & enc-depth, dec-depth & 2 \\  & dim-emb & 512 \\  & dim-rnn & 512 \\  & dropout-rnn & 0.1 \\  & o (optimizer) & adagrad \\  & mini-batch & 128 \\  & 1 (learning rate) & 0.01 \\ \hline Transformer & transformer-heads & 8 \\  & transformer-dim-ffn & 512 \\  & transformer-ffn-depth & 6 \\  & transformer-dropout & 0.1 \\  & label-smoothing & 0.1 \\  & optimizer & adam \\  & mini-batch & 128 \\ \hline \end{tabular}
\end{table} 
Table 6: Hyperparameter settings for the [PERSON] toolkit (SMT) [51], SRILM toolkit (SMT) [52], and the Marian toolkit (LSTM, transformer) [53]. \begin{table}
\begin{tabular}{l r r} \hline
**Dataset** & **Words** & **Types** \\ \hline GosVL & 142,762 & 20,729 \\ Gos & 826,532 & 57,424 \\ [PERSON]\_[PERSON] & 354,162 & 37,059 \\ [PERSON]\_[PERSON] & 274,897 & 28,472 \\ [PERSON]\_[PERSON] & 369,412 & 32,022 \\ All & 1,967,765 & 103,215 \\ \hline \end{tabular}
\end{table} 
Table 5: Sizes of datasets (number of words) and vocabularies (number of types) for each dataset.