# Parallel Bidirectionally Pretrained Taggers as Feature Generators
## 2 Materials and Methods
### Bidirectional Training  
Bidirectional training was popularized primarily via the phenomenon of Bidirectional Encoder Representations from Transformers (BERT) [25] and, later, its derivatives. While  
\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & **TreeTagger** & **spaCy** & **RNNTagger** & **Stanza** & **NLTK** & **Test Set** \\ \hline TreeTagger & & 6.15 & 6.81 & 6.40 & 7.79 & 4.25 \\ spaCy & 6.15 & & 6.09 & 3.35 & 10.14 & 5.19 \\ RNNTagger & 6.81 & 6.09 & & 5.27 & 5.82 & 6.84 \\ Stanza & 6.40 & 3.35 & 5.27 & & 9.93 & 5.71 \\ NLTK & 7.79 & 10.14 & 5.82 & 9.93 & & 9.05 \\ \hline average CEL & 6.79 & 6.43 & 6.00 & 6.24 & 8.42 & \\ test set CEL & 4.85 & 5.19 & 6.84 & 5.71 & **9.05** & \\ uniqueness & 1.94 & 1.24 & \(\mathbf{-0.84}\) & 0.53 & \(\mathbf{-0.63}\) & \\ UCA & 1164 & 538 & \(\mathbf{17}\) & 391 & \(\mathbf{0}\) & \\ \hline \hline \end{tabular}
\end{table}
Table 1: Cross-Entropy Loss (CEL) between probabilistic outputs of tagger pairs (top), and the average CEL, test set CEL, uniqueness score, and number of unique correct annotations (UCA) for each tagger with outlying values presented in bold (bottom).  
BERT uses bidirectional training for maximizing context awareness (using both left and right contexts in the creation of context-aware word embeddings), we used it to train separate, standalone models of our POS-taggers, which we later combined in a composite bidirectional architecture.  
To test if this is feasible, we trained three new taggers on the inverted dataset (generated simply through the inversion of our preannotated token array) with no other changes to the training procedures. The only difference in these inverted taggers (\(-\)I) was that they were supplied with inverted texts during tagging, and the obtained results needed to be inverted again to preserve the original token order. To tag the sentence _Hello world!_ using an inverted tagger, you need to supply it with the following list of tokens:  
1.!
2. world
3. Hello  
and simply invert the resulting array of tags or tag probabilities. As one of the goals was to test our hypothesis that these inverted taggers satisfy the composite environment requirements and perform well as standalone taggers, we recreated the tests from the previous section, and the results are presented in Table 2. The only addition is the bottom row, which shows the taggers' bidirectionally unique correct annotations (BUCA). Each time a tagger correctly annotates a token missed by its inverted counterpart (and vice-versa) it gains a BUCA point, and a higher BUCA score represents more effective bidirectional training of the tagger.  
We found that inverted taggers not only have actual unique contributions but, in some cases, outperform the standard taggers (spaCy-I against spaCy BUCA). Even in the case of TreeTagger, which had a very modest CEL of 1.54 (TreeTagger row, TreeTagger-I column of Table 2), UCAs were still found. Taggers also showed only small differences in the average CEL and test set CEL with no specific outliers. The highest recorded CEL against the test set was 6.05 from the Stanza-I tagger, which was still better than the results from NLTK and RNN-Tagger presented in the previous section (6.84 and 9.05, respectively). Finally, by comparing the UCAs between all six taggers (UCA row in Table 2), we found a tendency toward an equal dispersion, making each of these six taggers equally suitable for the composite environment.