# Explainable Identification of Similarities Between Entities for Discovery in Large Text
## 2 Automatic Similarity Analysis Method
### Weighted n-Gram Analysis for Identification of Similarities  
The analysis of identifiable similarities between documents of text is based on which identifying and assigning weights to n-grams that appear in both documents. The analysis receives two chunks of text data as its input, and identifies n-grams that are common between them. For that purpose, the words need to be converted to their stem form to identify similar n-grams. This is achieved through the SpaCy library, which provides mature automatic lemmatization of text. The concept of n-grams is needed since many patterns of similarities can not be detected through a single word. For instance, in a comparison of two preeminent scientists, a possible pattern of similarity between them is that both won a Nobel Prize. Since \"Nobel Prize\" is two words, the word \"Nobel\" and \"Prize\" will not be associated if each is used as a single word. Another example could be the term \"climate change\", where each of the two words has a meaning in its own, but when combined they reflect a different term. The use of n-grams can ensure that terms that are reflected by multiple words are also analyzed, and can be detected. The primary weakness of this technique is that the vast majority of matching n-grams in two documents are irrelevant to the similarities between the subjects discussed in these documents. That is, while the text might be similar due to common words that appear in both of them, the similarity of the text does not provide any useful insights to the user. For instance, simple words or phrases such as \"it is\" might appear numerous times in the documents, but the fact that they appear in both documents does not provide any useful insights. In that sense, the similarity between text documents provides similarity of the text, but not the similarity between the subjects that are being compared. To address that weakness, the analysis must be able to trim n-grams of common words that are frequent in both documents. Such n-grams are not expected to include useful insights, and will overwhelm the user with unimportant data, making it difficult to identify the true informative insights. That can be achieved by penalizing topics or words that are frequent in the English language. Algorithm 1 provides the high-resolution description of the method. ```
1:Input: doc1_data, doc2_data, eng_dict
2:Output: weighted_ngrams
3:Analyze n-grams in doc1_data and doc2_data
4:Identify common n-grams between documents
5:for each common n-gram do
6: Calculate weighted score based on frequency using eng_dict
7:endfor
8:Apply SpaCy for noun chunk and named entity recognition
9:Combine weighted scores of common n-grams from both documents
10:return weighted_ngrams
```  
**Algorithm 1** High-level algorithm for common n-grams analysis. Arguments
* doc1_data and doc2_data: N-gram data, noun chunks, and named entity information for each document. * eng_dict: A dictionary of English word frequencies for significance calculation. The backbone of the method is the weighting of the n-grams as they are evaluated for their ability to provide useful similarities between two text documents. Such weighting can be conceptualized as the identification of n-grams that are common to the two documents being compared, but are not so common in the general language used in these documents. That is, the weight of an n-gram increases when the n-gram is common in the two document, but decreases when the n-gram is common in the language of the document. That provides an explainable method that can identify specific n-grams that indicate on similarities between documents. In its most simple form, the weights can be determined using a basic frequency comparison:  
\[Weight=(f_{1}-f_{e})^{2}\times(f_{2}-f_{e})^{2}, \tag{1}\]  
where \(f_{1}\) and \(f_{2}\) represent n-gram frequencies in the two documents, and \(f_{e}\) represents baseline frequency for common words derived from a large and representative Corpus of Contemporary American English [50].