# Text Alignment in the Service of Text Reuse Detection
## 2 Related Work
### Distance Matrices  
SW and NW algorithms require the incorporation of a distance metric that measures the degree of similarity between tokens, followed by applying a predetermined threshold to determine their semantic alignment. The simplest comparative method employed by [PERSON] et al. [12] and [PERSON] et al. [14] assigns a score of 1 to identical tokens and 0 to tokens that do not match. However, applying such a metric to our corpus may not match tokens exhibiting orthographic variations. [PERSON] et al. [15] employed stemming processes before comparing tokens. Stemming involves heuristic methods to derive a word's root or stem form. However, stemming for morphologically rich languages is infeasible [16]. Extracting the lexeme form from each word requires a lemmatization process, which is performed using the part-of-speech (POS) tagging method [16]. [PERSON] et al. [17] designed a transition-based framework for morphosyntactic dependency parsing (termed _YAP_). This framework is a POS tagging model trained on modern Hebrew. We tested [17] on ancient Hebrew texts. The results detected the proper base form in 77% of the test cases. The relatively low accuracy is attributed to language changes over the centuries. Retraining _YAP_ on ancient languages like Hebrew is very limited due to the low availability of data [18]. We tested a more recent large language model published by [PERSON] et al. [19]. They fine-tuned a BERT model for the lemmatization task of Hebrew words, training it on modern Hebrew. As reported by [19], the model rarely predicts semantically related synonyms instead of the word's lexeme. We tested this model on rabbinic literature (Ancient Hebrew texts) and found that in 5% of the cases, the model predicted a synonym instead of the actual word's lexeme. The results detected the proper base form in 75% of the test cases. An alternative approach involves employing a distance metric based on the edit distance measure [8] to capture orthographic differences, as used by [PERSON] [20]. However, this method may also fail to match the conceptual distance between words that have undergone significant morphological alterations [5]. [PERSON] et al. [5] crafted a distance metric specifically tailored to Hebrew/Aramaic languages. They postulated a distance metric as a linear combination of two renowned distance metrics to encapsulate orthographic and semantic discrepancies. Initially, they employed a normalized version of the Levenshtein distance [8], quantifying the requisite character alterations to transmute one token into another. Subsequently, they utilized the cosine similarity between token vectors within a dense, low-dimensional word embedding space. For this purpose, ref. [5] harnessed the word2 vec neural network model [7] (termed W2V) trained on an extensive Hebrew corpus. An intrinsic limitation of the W2V model is its inability to proffer embeddings for out-of-vocabulary words [10]. This means that tokens absent during the training phase lack vector representations. To circumvent this limitation, ref. [5] modified their distance function to impose a gap penalty when a token lacks a vector representation. 'fastText' [10] is a word embedding methodology engineered to encapsulate word morphologies within a closely aligned vector space. As demonstrated by [21], fastText adeptly captures the intricacies of Hebrew morphology, reflecting the nuanced and multifaceted semantics inherent to the language. A distinguishing feature of fastText is its capability to vectorize entire terms and their respective n-grams during the model training phase. This capability empowers fastText to furnish vector representations even for out-of-vocabulary words [10]. Additionally, the mechanism of n-gram vectorization ensures that inflections of words are proximate within the vector space [10]. [PERSON] et al. [22] compared fastText with Word2 vec word embedding for Latin. fastText exceeded word2 vec by 6.9% to fetch similar lemmas.