# Parallel Bidirectionally Pretrained Taggers as Feature Generators
### Related Work  
The first comparative analysis of POS-taggers applied to texts in Serbian was conducted using 10-fold cross validation and compared three different taggers [15]. The study reported a high performance by both TnT (Trigrams'N'Tags) [16] and the TreeTagger [6] tool (accuracy on the training set, 93.86% and 91.78%, respectively), whilst SVMTool [3] was shown to have a somewhat better performance in certain special cases. The author also noted the possibility of combining different tagging methods and tools and integrating these with other NLP environments to open a wide area for further investigations and experiments regarding these solutions.  
Another study [7] experimented with three different taggers, namely the TreeTagger, TnT, and Unitex [17], for the purpose of annotating the Corpus of Contemporary Serbian (SrpKor2013) [18]. After an exhaustive evaluation and discussion, the author named TreeTagger as the best candidate and applied it for annotation of the SrpKor2013. This choice was justified by the fact that TreeTagger assigns the information about the lemma, which is an especially desired feature in the task of corpus tagging.  
We also recently compared several versions of TreeTagger and spaCy for Serbian [19] and found that each system has specific pros and cons. While spaCy's RNN-based contemporary approach to training gives it a slight edge when there is a high token overlap between the training and test sets, it underperforms compared with TreeTagger when dealing with unfamiliar words. When TreeTagger is in such a situation, a dictionary look-up is performed, but spaCy lacks such an instrument in its out-of-the-box form. The main question remaining is what is the best way to combine the pros of these taggers to get the optimal results.  
An experiment combining multiple POS-taggers for Swedish [20] concluded that most types of combination algorithms work better than the best standalone tagger. Another combination attempt was done for Icelandic [21], where the accuracy was improved by nearly two percent using multiple taggers and simple combination algorithms. These papers established a basic set of combining techniques. The baseline is _voting_, where the tag with the most votes from different taggers is picked. The problem with this approach is that it does not work at all when only two taggers are available, and even if there are more taggers available, there is a high chance of a tie occurring, which cannot be resolved without the implementation of another algorithm. A slight improvement is offered through _weighted voting_, which applies weights to different taggers or tags, so the chance of a tie is lowered. Other techniques, such as _bidding_ and _scoring_, require taggers to output tag probabilities. In _bidding_, the final tag is the one with the highest probability among all tags from all taggers, while in _scoring_, probabilities for tags are summed up among taggers, and the tag with the highest sum is selected.  
The latest milestone in the development of composite POS-taggers is the _stacking_ technique. This approach is used to eliminate the main cause of the low performance of composite taggers--the low-performing standalone tagger, offering high probabilities for its choice. This approach adds another supervised classifier on top of the results of standalone taggers, and if the classifier is well trained, it can learn that the low-performing tagger is not to be trusted. The initial experiment using probabilities as features [22] explored the composite system, which uses probabilistic outputs (for each tag and for each standalone tagger) of several standalone taggers as features for an HMM stacked classifier, which produces the final tag output.