# Text Semantic Annotation: A Distributed Methodology Based on Community Coherence
## 3 Materials and Methods
### Preprocessing  
In this work, we leverage the rich set of semantic information that can be exploited from Wikipedia manual annotations by authors as our training set for our mention universe. In general, this universe of mentions expands along with the semantic coverage of Wikipedia. However, our method supports knowledge extension by incorporating Wikipedia annotated corpus in our training set. A limitation in our knowledge acquisition methodology would lie in the semantic coverage of Wikipedia entities. However, due to the crowdsourcing nature of Wikipedia, the creation of the entities and the editing of the knowledge of diverse and commonly accepted textual descriptions by a large number of editors has established Wikipedia Entities as a standard for the task, in a process known as Wikification.  
As in [7; 8; 11; 12; 16] and several similar works, the set of mentions is extracted from an offline pre-processing of Wikipedia. Our methodology's universal set of entities consists of all Wikipedia articles in MediaWiki Main/Article namespace. A normalization needs to be carried out, so that all redirect pages are unified with their redirect target, including the necessary updates on Wikipedialinks to their target entity ids. Previously, this step involved following long redirect chains, however, this is remediated in recent Wikipedia snapshots.  
As in [8; 12; 16], we utilized inter-wiki anchors, entity titles, and redirect pages for extracting a set of anchors that can be linked to a Wikipedia Entity, using the XML dump of Wikipedia as a sole source of input, after a single parse of the dump file. Maintaining the _anchor id_, _mention entity id_, and _source article id_ as our input, and the corpus of Wikipedia dump, we efficiently can derive the tuple of available mentions (_anchor id_, _entity id_) including their occurrence prior-probability, also denoted as _commonness_, and the _anchor link probability_, denoted \(lp(a)\). We applied filtering rules for pruning some low absolute and relative frequency mentions, along with some rules for discarding unsuitable for annotation anchors, such as stop-words and characters, from the mention universe (i.e., mention vocabulary). This is a common practice of similar works [9; 10; 11; 12; 13; 14; 15; 16] that aims to maintain semantically significant mentions, while removing the unnecessary noise.  
Specifically, the mentions dictionary contains roughly 37.5 million distinct anchors, containing 19.3 million entities which are interconnected in a link-graph of about 1.11 billion edges including edge enhancement via entity title redirect page processing.  
As a next preprocessing step, we employ either Louvain or Clauset-Newman-Moore algorithm community detection on the Wikipedia Entity graph to achieve segmentation based on Wikipedia Entity community coherence characteristics. Hence, leveraging the ensured concept consistency, we manage to focus on a strict subset of Wikipedia Entities and thus improve the overall disambiguation efficiency by substantially demoting unmeaningful context from the collective mention consensus. Even if the computational complexity of these algorithms is O(n log\({}^{2}\)n), as outlined in [2; 3] both algorithms' performance was evaluated in the context of the problem.  
This offline task is required for the initialization process, as online updates can be applied on the fly leveraging a differential changes stream. Accordingly, for the community detection step, similar techniques may be applicable for on-the-fly community classification and rebalancing.