# Text Semantic Annotation: A Distributed Methodology Based on Community Coherence
## 4 Results  
Our experimental evaluation was carried out using entities from Wikimedia Wikipedia dumps from 2019-12-20 [23]. Several raw database tables are available; however, we used the full copy of all Wikimedia pages enwiki-20191220-pages-articles [23], in the wikitext format of source and metadata embedded in XML as a single source of truth throughout our assessment. The efficient pre-processing of Wikipedia at its current size of 19.3 million entities interconnected in a link-graph of about 1.11 billion edges has been a challenging exercise. However, the employment of big-data techniques was critical for streamlining the ETL process. Due to the crowdsourcing nature of Wikipedia, there is a strong tendency for growth in the dataset over time, although our implementation's inherent distributed architecture and processing model would be capable of handling this big-data problem at a far larger scale. For the experimental analysis, we focused on the established wiki-disamb30 dataset from [12] for evaluating the disambiguation process. This dataset consists of roughly 1.4 million short text segments randomly derived from Wikipedia pages and is generally considered as a baseline copious similar works [12; 15; 16; 21]. Each text segment comprises approximately 30 words, including at least one ambiguous anchor and its respective Wikipedia article title annotation. As Wikipedia changes  
Figure 1: The proposed methodology flowchart. over time, some preliminary pre-processing was required for either updating or filtering deprecated Wikipedia entities. Our experiments involved a 64 vCPU and 240 GB memory Google Cloud Platform setup, leveraging the scalability and parallelism of the Apache Spark framework during the ETL and pre-processing phase from the pages-articles.xml dump file. Robust implementations of Clauset-Newman-Moore [24] and Louvain [25] algorithms were leveraged for the two variants of our proposed methodologies. We adopted the sclearn.GaussianProcessClassifier [26] for our Gaussian Process Classification step. At this point we should broadly emphasize the exceptionally vast context variability. Nevertheless, as the experimentation showed, the community detection (regardless of the algorithm applied) managed to efficiently partition the dataset into meaningful communities, with divergent community structures per case. Regarding the disambiguation performance comparison, the classic prediction performance metrics are considered. Specifically, in reference to the entity linking concept, those can be outlined as shown below:  
* Precision: The fraction of the correctly annotated anchors over the valid mentions. \[\text{Precision}=\frac{TP}{TP+FP}\] (5)
* Recall: The proportion of actual correctly annotated anchors. \[\text{Precision}=\frac{TP}{\left|\text{{mentions}}\right|}\] (6)
* F1 Score: the harmonic mean of precision and recall. \[\text{F1}=2*\frac{Precision*Recall}{Precision+Recall}\] (7)  
The assessment of the proposed methodology variants during an initial exploratory process contributed considerable insights. Similarly, the distinct community detection algorithms included in this work were more promising for further large-scale evaluation. We focused on the analysis of the entity linking disambiguation step, outlining the value of our _community-coherence_ based methodology, and the review of the introduced confidence score as an indication of the disambiguation quality. Our experiments involved the evaluation of our described methodologies, in two variants. The first variant is based on the Louvain algorithm, and the second on the Clauset-Newman-Moore algorithm for the community detection step. As a comparative baseline, we included the performance of TAGME [12], along with the performance of our Gaussian Process Classifier methodology, without the consideration of the _community coherence_ feature, to outline the contribution of the feature to the overall approach.