Package: pytextdist
python-text-distanceA python implementation of a variety of text distance and similarity metrics.InstallHow to UseModuleEdit DistanceLevenshtein Distance & SimilarityLongest Common Subsequence Distance & SimilarityDamerau-Levenshtein Distance & SimilarityHamming Distance & SimilarityJaro & Jaro-Winkler Similarity & SimilarityVector SimilarityCosine SimilarityJaccard SimilaritySorensen Dice SimilarityQ-Gram SimilarityCustomize PreprocessInstallRequirements:py>=3.3Install Command:pip install pytextdistHow to useThe functions in this package takes two strings as input and return the distance/similarity metric between them. The preprocessing of the strings are included in the functions with default recommendation. If you want to change the preprocessing seeCustomize Preprocessing.ModulesEdit DistanceBy default functions in this module consider single character as the unit for editting.Levenshtein Distance & Similarity: edit with insertion, deletion, and substitutionfrompytextdist.edit_distanceimportlevenshtein_distance,levenshtein_similaritystr_a='kitten'str_b='sitting'dist=levenshtein_distance(str_a,str_b)simi=levenshtein_similarity(str_a,str_b)print(f"Levenshtein Distance:{dist:.0f}\nLevenshtein Similarity:{simi:.2f}")>>LevenshteinDistance:3>>LevenshteinSimilarity:0.57Longest Common Subsequence Distance & Similarity: edit with insertion and deletionfrompytextdist.edit_distanceimportlcs_distance,lcs_similaritystr_a='kitten'str_b='sitting'dist=lcs_distance(str_a,str_b)simi=lcs_similarity(str_a,str_b)print(f"LCS Distance:{dist:.0f}\nLCS Similarity:{simi:.2f}")>>LCSDistance:5>>LCSSimilarity:0.62Damerau-Levenshtein Distance & Similarity: edit with insertion, deletion, substitution, and transposition of two adjacent unitsfrompytextdist.edit_distanceimportdamerau_levenshtein_distance,damerau_levenshtein_similaritystr_a='kitten'str_b='sitting'dist=damerau_levenshtein_distance(str_a,str_b)simi=damerau_levenshtein_similarity(str_a,str_b)print(f"Damerau-Levenshtein Distance:{dist:.0f}\nDamerau-Levenshtein Similarity:{simi:.2f}")>>Damerau-LevenshteinDistance:3>>Damerau-LevenshteinSimilarity:0.57Hamming Distance & Similarity: edit with substition; note that hamming metric only works for phrases of the same lengthsfrompytextdist.edit_distanceimporthamming_distance,hamming_similaritystr_a='kittens'str_b='sitting'dist=hamming_distance(str_a,str_b)simi=hamming_similarity(str_a,str_b)print(f"Hamming Distance:{dist:.0f}\nHamming Similarity:{simi:.2f}")>>HammingDistance:3>>HammingSimilarity:0.57Jaro & Jaro-Winkler Similarity: edit with transpositionfrompytextdist.edit_distanceimportjaro_similarity,jaro_winkler_similaritystr_a='sitten'str_b='sitting'simi_j=jaro_similarity(str_a,str_b)simi_jw=jaro_winkler_similarity(str_a,str_b)print(f"Jaro Similarity:{simi_j:.2f}\nJaro-Winkler Similarity:{simi_jw:.2f}")>>JaroSimilarity:0.85>>Jaro-WinklerSimilarity:0.91Vector SimilarityBy default functions in this module use unigram (single word) to build vectors and calculate similarity. You can changento other numbers for n-gram (n contiguous words) vector similarity.Cosine Similarityfrompytextdist.vector_similarityimportcosine_similarityphrase_a='For Paperwork Reduction Act Notice, see your tax return instructions. For Paperwork Reduction Act Notice, see your tax return instructions.'phrase_b='For Disclosure, Privacy Act, and Paperwork Reduction Act Notice, see separate instructions. Form 1040'simi_1=cosine_similarity(phrase_a,phrase_b,n=1)simi_2=cosine_similarity(phrase_a,phrase_b,n=2)print(f"Unigram Cosine Similarity:{simi_1:.2f}\nBigram Cosine Similarity:{simi_2:.2f}")>>UnigramCosineSimilarity:0.65>>BigramCosineSimilarity:0.38Jaccard Similarityfrompytextdist.vector_similarityimportjaccard_similarityphrase_a='For Paperwork Reduction Act Notice, see your tax return instructions. For Paperwork Reduction Act Notice, see your tax return instructions.'phrase_b='For Disclosure, Privacy Act, and Paperwork Reduction Act Notice, see separate instructions. Form 1040'simi_1=jaccard_similarity(phrase_a,phrase_b,n=1)simi_2=jaccard_similarity(phrase_a,phrase_b,n=2)print(f"Unigram Jaccard Similarity:{simi_1:.2f}\nBigram Jaccard Similarity:{simi_2:.2f}")>>UnigramJaccardSimilarity:0.47>>BigramJaccardSimilarity:0.22Sorensen Dice Similarityfrompytextdist.vector_similarityimportsorensen_dice_similarityphrase_a='For Paperwork Reduction Act Notice, see your tax return instructions. For Paperwork Reduction Act Notice, see your tax return instructions.'phrase_b='For Disclosure, Privacy Act, and Paperwork Reduction Act Notice, see separate instructions. Form 1040'simi_1=sorensen_dice_similarity(phrase_a,phrase_b,n=1)simi_2=sorensen_dice_similarity(phrase_a,phrase_b,n=2)print(f"Unigram Sorensen Dice Similarity:{simi_1:.2f}\nBigram Sorensen Dice Similarity:{simi_2:.2f}")>>UnigramSorensenDiceSimilarity:0.64>>BigramSorensenDiceSimilarity:0.36Q-Gram Similarityfrompytextdist.vector_similarityimportqgram_similarityphrase_a='For Paperwork Reduction Act Notice, see your tax return instructions. For Paperwork Reduction Act Notice, see your tax return instructions.'phrase_b='For Disclosure, Privacy Act, and Paperwork Reduction Act Notice, see separate instructions. Form 1040'simi_1=qgram_similarity(phrase_a,phrase_b,n=1)simi_2=qgram_similarity(phrase_a,phrase_b,n=2)print(f"Unigram Q-Gram Similarity:{simi_1:.2f}\nBigram Q-Gram Similarity:{simi_2:.2f}")>>UnigramQ-GramSimilarity:0.32>>BigramQ-GramSimilarity:0.15Customize PreprocessingAll functions will performpytextdist.preprocessing.phrase_preprocessingto clean the input strings and convert them to a list of tokens.When grain="char" - remove specific characters from the string and convert it to a list of charactersThe following boolean parameters control what characters to remove/change from the string (all True by default):- ignore_non_alnumspc: whether to remove all non-numeric/alpha/space characters- ignore_space: whether to remove all space- ignore_numeric: whether to remove all numeric characters- ignore_case: whether to convert all alpha charachers to lower caseExample:frompytextdist.preprocessingimportphrase_preprocessingbefore='AI Top-50'after=phrase_preprocessing(before,grain='char')print(after)>>['a','i','t','o','p']When grain="word" - convert the string to a list of words and remove specific characters from the wordsThe string is firstly converted to a list of words assuming all words are separated by one space, then the following boolean parameters control what characters to remove/change from the string (all True by default):- ignore_non_alnumspc: whether to remove all non-numeric/alpha/space characters- ignore_numeric: whether to remove all numeric characters- ignore_case: whether to convert all alpha charachers to lower caseExample:frompytextdist.preprocessingimportphrase_preprocessingbefore='AI Top-50'after=phrase_preprocessing(before,grain='word')print(after)>>['ai','top']Functions under the vector similarity module will also performpytextdist.preprocessing.ngram_counteron the list return frompytextdist.preprocessing.phrase_preprocessing.Convert a list of tokens to a counter of the n-gramsThe following parameter control the n to use for n-grams (1 by default):- n: number of contiguous items to use to form a sequenceExample:frompytextdist.preprocessingimportphrase_preprocessing,ngram_counterbefore='AI Top-50 Company'after=phrase_preprocessing(before,grain='word')print(after)ngram_cnt_1=ngram_counter(after,n=1)print(ngram_cnt_1)ngram_cnt_2=ngram_counter(after,n=2)print(ngram_cnt_2)>>['ai','top','company']>>Counter({'ai':1,'top':1,'company':1})>>Counter({'ai top':1,'top company':1})