## 3 Work-Flow
### From Unstructured Text to Semantic Flows  
Figure 1, step 2, shows the first text-processing step of NLP tools to normalize input words into tokens. The SNAP framework relies on the linguistic models that are native to the NLP framework used, and SNAP does not evolve or augment these models. SNAP currently uses Natural Language Toolkit (NLTK) for NLP processing, but it can be swapped for any other NLP framework [40]. For a detailed description and implementation of various NLP algorithms that process the raw text, please see the NLP framework documentation [40]. In this section, we will not cover the algorithmic details implemented by different NLP engines since the same NLP task can be implemented in multiple ways from dictionary look-up, stochastic search algorithms, or models built using machine learning. It is important to note, though, that the framework is designed to accommodate various NLP tool kits with varying algorithm implementations which will result in slightly different tokens. The input text is first tokenized, which splits the stream of characters in the raw text file into groups that roughly delineate individual words. The tokenized words are stemmed to remove various inflected forms of the same word so all word variations can be analyzed as a single token. For example, the original word \"identified\" is stemmed into the present tense _identify_, noun plurals are removed \"whales\" \(\rightarrow\)_whale_, and so forth. Each processed document is split into sentences that are then tagged for part-of-speech (POS). The words tagged by the same POS tag that span more than one word are combined into a single token. For example, \"16 year old\" is concatenated to _16 yearold_, geographic and organization entities are also merged to a single token \"Wildlife Service\" \(\rightarrow\)_wildlifservice_, \"human society\" \(\rightarrow\)_humansciety_ and so forth. The words that were not combined by a POS tagging are lemmatized to produce word roots. Examples of this processing step include converting words such as \"observations,\" \"'gloomy,\" \"determination\" to _observ, gloom_ and _determin_ respectively. The lemmatization is an essential step that reduces the many word-forms used to a single token and seeds the notion of a semantic concept; for example \"domination,\" \"dominate,\" \"dominator,\" and \"domineering\" are all lemmatized to a single token _dominat_. The final step of the word-to-token conversion is the named entity recognition (NER) that will also combine multiple words into a single token. These examples include \"Bird Ridge\" \(\rightarrow\)_BirdRidge_, \"[PERSON] [PERSON]\" \(\rightarrow\)_senatornerkeley_, \"United States Of America\" \(\rightarrow\)_unitedstatesofamerica_. Figure 1, step 3, illustrates the dictionary-based stop-word removal that removes tokens from the document that do not have a semantic meaning (e.g., \"the,\" \"from,\" \"on\" etc.). Additional tokens are removed based on the user-defined preference using lower and upper percentiles thresholds for the document's token frequencies (TF). The resulting set of tokens \(T_{d}\) in a document \(d\) is used to construct a graph \(G_{d}\) representing the document's semantic network of the tokens and their co-occurrences in a sentence. The semantic network generation using lexical chaining with sliding window size of three tokens is shown in Figure 1, step 4. Each unique token is represented as a network node. The undirected network edges are generated by creating connections between all pairs of tokens in a sentence within the fixed window size. The illustration shows sub-graphs generated for consecutive positions of the sliding windows at the top of the pane, while the bottom of the pane shows the composite graph that merged the constituent sub-graphs into a single graph. A semantic network is generated for each document by applying sub-graph generation for every sentence with the sliding window shifted by one token and creating a composite graph. Figure 1: Illustration of SNAP’s work-flow.