# Explainable Identification of Similarities Between Entities for Discovery in Large Text
## 2 Automatic Similarity Analysis Method
### Weighted n-Gram Analysis for Identification of Similarities
The weight can therefore be conceptualized as the frequency of the word in the document in which it appears compared to its frequency in the English language. Words that are more frequent in the document are assigned with a higher weight, but that weight is also reduced when the word is more frequent in the English language in general. To allow adjusting the noise level through the weighting, a logarithmic scaling was applied as shown by Equation (2). The logarithmic scaling provided a smoother weighting that further lowered the weight of n-grams that are common in the English language, therefore reducing the amount of similarity insights that reflect the use of language rather than similarity insights that are relevant to the user. \[Weight=f_{1}\times\log\left(\frac{f_{1}}{f_{e}}\right)+f_{2}\times\log\left( \frac{f_{2}}{f_{e}}\right) \tag{2}\]  
The weights can be refined with thresholding based on average frequency. That is, if the weight is smaller than the threshold, the weight is set to zero. The intuition of the rule is that if the word is more frequent in the English language than it is in any of these documents, that word might not be able to provide useful insights about these documents. The thresholding is required to eliminate common terms that were not meaningful in the distinction across documents. \[threshold =\min(f_{1},f_{2},f_{e}) \tag{3}\] \[weight =\frac{\log(f_{1}+1)\times e^{f_{2}}}{\sqrt{f_{e}+1}}\] (4) \[iff_{e}>threshold:\,weight=0 \tag{5}\]  
The final formula combines frequency differences, entity presence, and noun chunk recognition for an enriched analysis:  
\[Weight=(\Delta f_{1}+\Delta f_{2})\times BaseMultiplier \tag{6}\]  
where:  
* Base multiplier = \(1+0.5\times\) (entity presence + noun chunk presence)\(\Delta f_{1}\) and \(\Delta f_{2}\) are the squared differences between the n-gram's frequency in documents 1 and 2 compared to the English corpus frequency: \[\Delta f_{1} =(f_{1}-f_{e})^{2}\times Normalization\ Factor\] \[\Delta f_{2} =(f_{2}-f_{e})^{2}\times Normalization\ Factor\]
* Entity presence is 0.5 if the n-gram is a named entity. * Noun chunk presence is 0.5 if the n-gram is a noun chunk. This formula synthesizes linguistic features and frequency adjustments to provide a more meaningful representation of textual similarity. If the n-gram contains an entity or noun chunk, the weight assigned to the n-gram will be increased. Rather than relying on raw counts alone, it makes the comparison meaningful beyond mere frequency matching, allowing to prioritize n-grams that are more likely to be meaningful to the user. Figure 2 illustrates the steps of the analysis. The intuition of the method is that it identifies topics that are common to the two text documents, but are not common in the English language. That means that the formula assigns higher weights to words that appear more frequently in both documents, but penalizes words that are frequent in other documents that are not related to the documents or topics that are being compared. The frequency of words in other documents is determined by their frequency in the English language. For instance, words such as \"is\" or \"the\" that are common between virtually any two English documents will not be assigned with high weights by the method since these words are also common in the English language in general. That leads to high weights assigned to topics that are relevant to the specific similarities between the two documents. Words that are less common in the English language but appear in both documents in relatively high frequency will be assigned higher weights, and therefore are likely to be identified. Because the formula is deterministic, and the reason for the weight assigned to each word is clear, the model is explainable. That is different from methods based on deep neural networks such as large language models, where the rules are highly complex and non-intuitive, and users or developers of these methods cannot fully understand how the output is generated.