Question: Were the two pre-trained vectors extracted from the same source in the first experiment?
Evidence: In the first experiment the model was trained using pretrained vectors extracted from two different sources, Common Crawl and Wikipedia and can be found at the official FastText web page BIBREF8. Both sources were trained on the same algorithm called FastText BIBREF9, an extension of Word2Vec that treats tokens as an average sum of sub-words and finds similarities of words based on their n-grams. The configuration of the FastText model for Wikipedia vectors is according to BIBREF10, whilst the model for CC vectors is a position-weight CBOW 5 length n-grams with a window size of 5 tokens and 10 negative words. The file with the Common Crawl vectors consists of 2.000.000 tokens with 300 dimension, whereas the file with the Wikipedia vectors consists of 300.000 tokens with 300 dimension.The results can be viewed in the following table, with the first part describing the Common Crawl results and the second one the Wikipedia results.
At the results, POS and morph classes refer to the tag labels explained in SECREF4, whilst only POS classes relate to annotated labels that describe only the part of speech. It is evident that even though the CC vectors are noisy, coming from a web source, they lead to better results than Wikipedia, possibly because they have a larger variety of tokens.
In the next experiment, the dataset was used for the composition of embeddings for the part of speech tagger. The dataset was trained on a FastText model with the same parameters that extracted the Common Crawl vectors. As a result, 140.000 vectors with 300 dimension were exported. It must be mentioned that the tagset with the morphological features was used.
The values of the metrics in this case were almost as good and comparable to the CC ones. However, the model trained with a larger vocabulary had higher results. Also, the model with the dataset vectors did not have the flexibility to classify unknown words.
As a next step, the test set of the dataset was altered by replacing words with syntactical mistakes to test the tolerance of the model in OOV words. Suffixes of verbs were altered and vowels were replaced with others, affecting 20% of the tokens of the dataset. Using again the more complex tagset for training, the results can be found in Table 3.
What can be concluded is that the model did not have a flexibility in OOV words. Of course, this can also be an advantage, meaning that the model recognized the mismatch of a wrong word with its class.
One disadvantage that the previous model had is that for unknown words the model assigned a zero vector, affecting the testing results. In order to minimize this problem, the unknown words were first passed through a FastText model to get a vector from their subwords. The resulting vectors were imported in the vocabulary with the CC vectors before training. The model was also trained using as a vocabulary the unknown words and the tokens from the Common Crawl vectors, both buffered in the same FastText model. Results are listed in Table 4.
It was noticed that the model performed better when using the vectors from different FastText models. It was expected that the second experiment would have performed better, as the tokens were inserted into the same FastText model and the vectors exported from both sources should match.
Both datasets were fed into the library in proper format for training. In training process, the entity recognizer had the same configuration with the POS tagger, using the same percentages for train, validation and test sets. It must be noted that all the models used the Common Crawl pretrained vectors for a vocabulary. The results are compared using the macro F1 score.
At first the datasets from both sources (Makedonia, Wikipedia) were used for training with 10 iterations and testing from the model. The results can be viewed in the following table:
It seemed that the average F1 score was higher for the Makedonia corpus, as it was the basis of the configuration for the keyword list. In order to have an objective evaluation, the results of each corpus per entity class were observed.
Both sources had good results in non entity tokens, which affected the F1 score. Moreover, the model did not perform well for facilities, as polyglot's Greek recognizer does not support that class and FAC entities cover a small amount of the list.
In the second experiment, the datasets were compared to a common test set that followed the desired set of rules.
Again, the Makedonia corpus performed better, because of the proper annotation on the keyword list.
In an experiment worth mentioning the correlation of the part of speech with the performance of the recognizer was explored. In this experiment, both pipelines (part of speech, entity recognition) were used for training with 30 iterations and the model was trained twice: with and without the usage of the part of speech information for recognition.
It is evident that the recognizer did not gain knowledge from the part of speech tags of the tokens.
In the first experiment the model was trained using pretrained vectors extracted from two different sources, Common Crawl and Wikipedia and can be found at the official FastText web page [9].
Answer: No, they are from Common Crawl and Wikipedia respectively.