Package: weighted-levenshtein
Use CasesMost existing Levenshtein libraries are not very flexible: all edit operations have cost 1.However, sometimes not all edits are created equal. For instance, if you are doing OCR correction, maybe substituting ‘0’ for ‘O’ should have a smaller cost than substituting ‘X’ for ‘O’. If you are doing human typo correction, maybe substituting ‘X’ for ‘Z’ should have a smaller cost, since they are located next to each other on a QWERTY keyboard.This library supports all theses use cases, by allowing the user to specify different weights for edit operations involving every possible combination of letters. The core algorithms are written in Cython, which means they are blazing fast to run.The Levenshtein distance function supports setting different costs for inserting characters, deleting characters, and substituting characters. Thus, Levenshtein distance is well suited for detecting OCR errors.The Damerau-Levenshtein distance function supports setting different costs for inserting characters, deleting characters, substituting characters, and transposing characters. Thus, Damerau-Levenshtein distance is well suited for detecting human typos, since humans are likely to make transposition errors, while OCR is not.More InformationLevenshtein distance:https://en.wikipedia.org/wiki/Levenshtein_distanceandhttps://en.wikipedia.org/wiki/Wagner%E2%80%93Fischer_algorithmOptimal String Alignment:https://en.wikipedia.org/wiki/Damerau%E2%80%93Levenshtein_distance#Optimal_string_alignment_distanceDamerau-Levenshtein distance:https://en.wikipedia.org/wiki/Damerau%E2%80%93Levenshtein_distance#Distance_with_adjacent_transpositionsInstallationpip installweighted-levenshteinUsage Exampleimportnumpyasnpfromweighted_levenshteinimportlev,osa,dam_levinsert_costs=np.ones(128,dtype=np.float64)# make an array of all 1's of size 128, the number of ASCII charactersinsert_costs[ord('D')]=1.5# make inserting the character 'D' have cost 1.5 (instead of 1)# you can just specify the insertion costs# delete_costs and substitute_costs default to 1 for all characters if unspecifiedprint(lev('BANANAS','BANDANAS',insert_costs=insert_costs))# prints '1.5'delete_costs=np.ones(128,dtype=np.float64)delete_costs[ord('S')]=0.5# make deleting the character 'S' have cost 0.5 (instead of 1)# or you can specify both insertion and deletion costs (though in this case insertion costs don't matter)print(lev('BANANAS','BANANA',insert_costs=insert_costs,delete_costs=delete_costs))# prints '0.5'substitute_costs=np.ones((128,128),dtype=np.float64)# make a 2D array of 1'ssubstitute_costs[ord('H'),ord('B')]=1.25# make substituting 'H' for 'B' cost 1.25print(lev('HANANA','BANANA',substitute_costs=substitute_costs))# prints '1.25'# it's not symmetrical! in this case, it is substituting 'B' for 'H'print(lev('BANANA','HANANA',substitute_costs=substitute_costs))# prints '1'# to make it symmetrical, you need to set both costs in the 2D arraysubstitute_costs[ord('B'),ord('H')]=1.25# make substituting 'B' for 'H' cost 1.25 as wellprint(lev('BANANA','HANANA',substitute_costs=substitute_costs))# now it prints '1.25'transpose_costs=np.ones((128,128),dtype=np.float64)transpose_costs[ord('A'),ord('B')]=0.75# make swapping 'A' for 'B' cost 0.75# note: now using dam_lev. lev does not support swapping, but osa and dam_lev do.# See Wikipedia links for difference between osa and dam_levprint(dam_lev('ABNANA','BANANA',transpose_costs=transpose_costs))# prints '0.75'# like substitution, transposition is not symmetrical either!print(dam_lev('BANANA','ABNANA',transpose_costs=transpose_costs))# prints '1'# you need to explicitly set the other direction as welltranspose_costs[ord('B'),ord('A')]=0.75# make swapping 'B' for 'A' cost 0.75print(dam_lev('BANANA','ABNANA',transpose_costs=transpose_costs))# now it prints '0.75'lev,osa, anddam_levare aliases forlevenshtein,optimal_string_alignment, anddamerau_levenshtein, respectively.Detailed Documentationhttp://weighted-levenshtein.readthedocs.io/Important NotesAll string lookups are case sensitive.The costs parameters only accept numpy arrays, since the underlying Cython implementation relies on this for fast lookups. The numpy arrays are indexed using theord()value of the characters. Thus, only the first 128 ASCII letters are accepted, anddictandlistare not accepted. Consequently, the strings must be strictlystrobjects, notunicode.This library is compatible with both Python 2 and Python 3 (seetox.inifor tested versions).Use as Cython libraryfromweighted_levenshtein.clevcimportc_levenshteinaslev,c_optimal_string_alignmentasosa,c_damerau_levenshteinasdam_levimportnumpyasnpa=np.ones(128,dtype=np.float64)b=np.ones((128,128),dtype=np.float64)print(lev("BANANA",4,"BANANAS",5,a,a,b))For the Cython API, functions are prefixed with ac_with respect to the Python API. Also, the string parameters are followed by their length. The data types of the numpy arrays specifying the costs still need to benp.float64, consistent with the Python API.Function signatures below:cdefdoublec_damerau_levenshtein(unsignedchar*str_a,Py_ssize_tlen_a,unsignedchar*str_b,Py_ssize_tlen_b,double[::1]insert_costs,double[::1]delete_costs,double[:,::1]substitute_costs,double[:,::1]transpose_costs)nogilcdefdoublec_optimal_string_alignment(unsignedchar*word_m,Py_ssize_tm,unsignedchar*word_n,Py_ssize_tn,double[::1]insert_costs,double[::1]delete_costs,double[:,::1]substitute_costs,double[:,::1]transpose_costs)nogilcdefdoublec_levenshtein(unsignedchar*word_m,Py_ssize_tm,unsignedchar*word_n,Py_ssize_tn,double[::1]insert_costs,double[::1]delete_costs,double[:,::1]substitute_costs)nogil