# Parallel Bidirectionally Pretrained Taggers as Feature Generators
## 2 Materials and Methods
### Training and Tagging Pipelines  
For the purposes of this research, we devised one algorithm for bidirectional training of the proposed stacked architecture and another for tagging documents using the trained resources. The training pipeline is composed of the bidirectional training of the standalone \begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & **TreeTagger** & **spaCy** & **Stanza** & **TreeTagger-I** & **spaCy-I** & **Stanza-I** \\ \hline TreeTagger & & 6.15 & 6.40 & **1.54** & 6.18 & 6.27 \\ spaCy & 6.15 & & 3.35 & 6.41 & 4.12 & 4.00 \\ Stanza & 6.40 & 3.35 & & 6.53 & 4.35 & 3.19 \\ TreeTagger-I & **1.54** & 6.41 & 6.53 & & 6.43 & 6.18 \\ spaCy-I & 6.18 & 4.12 & 4.35 & 6.43 & & 5.06 \\ Stanza-I & 6.27 & 4.00 & 3.19 & 6.18 & 5.06 & \\ \hline Average CEL & 5.31 & 4.81 & 4.76 & 5.42 & 5.23 & 4.94 \\ test set CEL & 4.25 & 5.19 & 5.71 & 4.58 & 5.17 & **6.05** \\ uniqueness & 1.06 & \(-\)0.38 & \(-\)0.95 & 0.84 & 0.06 & \(-\)1.31 \\ UCA & 57 & 59 & 57 & 42 & 59 & 48 \\ BUCA & 444 & 1038 & 1002 & 394 & 1064 & 695 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: CEL between tagger pairs, including the inverted taggers (top), average CEL, test set CEL, uniqueness score, and UCA and BUCA for each tagger (bottom) with some outlying values presented in bold. taggers and the fine-tuning of a stacked classifier using their respective probabilistic outputs on a token-level. This algorithm is depicted in Figure 2 and can be divided into four steps. (1) Training and tuning set formation. First, we group annotated sentences into two chunks (we shuffled them beforehand and used a four to one ratio, although this is not obligatory). We used the bigger chunk as the training set and the smaller chunk as a tuning set, splitting the data vertically into two separate arrays: one containing only tokens and the other containing correct POS tags for those tokens. The first step resulted in three arrays: the training set containing both tokens and POS tags; one tuning set containing only tokens; and the other tuning set containing only the POS tags. (2) Standalone tagger training. We used the previously built training set to train the selected standalone taggers. The architecture is vertically scalable, as presented in Figure 2, where...* denotes any number of additional taggers, with the only requirement being that they produce probabilistic outputs. We used the TreeTagger, spaCy, and Stanza POS-taggers for the experiment and trained them bidirectionally with standard out-of-the-box training for each tagger. Taggers do not need to use the same tagset, as every tag for every tagger becomes a new, separate feature (note, however, that this will render their standalone evaluation for resulting tagset pointless). Basically, the second step results in standalone tagger models trained on the bigger portion of the annotated dataset. (3) Creation of the probability tensor. This tensor was formed using the tuning token array created in step 1 and the pretrained standalone taggers created in step 2. It should be noted that it is also possible to use other pretrained taggers, but they should be trained on the set not overlapping the tuning set to avoid bias. We took probabilities for each tag from each tagger used (inverted taggers tagging the inverted array of tokens) and for each token in the tuning array (by applying the SoftMax function to normalize outputs of RNN-based  
Figure 2: Training algorithm for the composite model with the right side (connected via dashed line) being the bidirectional expansion. The architecture is vertically scalable (uncontainable) with the addition of any number of new taggers bidirectionally—represented with...*. taggers). These probability matrices (for each token and for each tag) for each standalone tagger were concatenated into a single tensor object, the output of this step. (4) Stacked classifier training. Stacked classifier training was performed using the tuning tag array created in the first step and the probability tensor created in the previous step.