Package: textdistance
TextDistanceTextDistance-- python library for comparing distance between two or more sequences by many algorithms.Features:30+ algorithmsPure python implementationSimple usageMore than two sequences comparingSome algorithms have more than one implementation in one class.Optional numpy usage for maximum speed.AlgorithmsEdit basedAlgorithmClassFunctionsHammingHamminghammingMLIPNSMlipnsmlipnsLevenshteinLevenshteinlevenshteinDamerau-LevenshteinDamerauLevenshteindamerau_levenshteinJaro-WinklerJaroWinklerjaro_winkler,jaroStrcmp95StrCmp95strcmp95Needleman-WunschNeedlemanWunschneedleman_wunschGotohGotohgotohSmith-WatermanSmithWatermansmith_watermanToken basedAlgorithmClassFunctionsJaccard indexJaccardjaccardSørensen–Dice coefficientSorensensorensen,sorensen_dice,diceTversky indexTverskytverskyOverlap coefficientOverlapoverlapTanimoto distanceTanimototanimotoCosine similarityCosinecosineMonge-ElkanMongeElkanmonge_elkanBag distanceBagbagSequence basedAlgorithmClassFunctionslongest common subsequence similarityLCSSeqlcsseqlongest common substring similarityLCSStrlcsstrRatcliff-Obershelp similarityRatcliffObershelpratcliff_obershelpCompression basedNormalized compression distancewith different compression algorithms.Classic compression algorithms:AlgorithmClassFunctionArithmetic codingArithNCDarith_ncdRLERLENCDrle_ncdBWT RLEBWTRLENCDbwtrle_ncdNormal compression algorithms:AlgorithmClassFunctionSquare RootSqrtNCDsqrt_ncdEntropyEntropyNCDentropy_ncdWork in progress algorithms that compare two strings as array of bits:AlgorithmClassFunctionBZ2BZ2NCDbz2_ncdLZMALZMANCDlzma_ncdZLibZLIBNCDzlib_ncdSeeblog postfor more details about NCD.PhoneticAlgorithmClassFunctionsMRAMRAmraEditexEditexeditexSimpleAlgorithmClassFunctionsPrefix similarityPrefixprefixPostfix similarityPostfixpostfixLength distanceLengthlengthIdentity similarityIdentityidentityMatrix similarityMatrixmatrixInstallationStableOnly pure python implementation:pipinstalltextdistanceWith extra libraries for maximum speed:pipinstall"textdistance[extras]"With all libraries (required forbenchmarkingandtesting):pipinstall"textdistance[benchmark]"With algorithm specific extras:pipinstall"textdistance[Hamming]"Algorithms with available extras:DamerauLevenshtein,Hamming,Jaro,JaroWinkler,Levenshtein.DevVia pip:pipinstall-egit+https://github.com/life4/textdistance.git#egg=textdistanceOr clone repo and install with some extras:gitclonehttps://github.com/life4/textdistance.git
pipinstall-e".[benchmark]"UsageAll algorithms have 2 interfaces:Class with algorithm-specific params for customizing.Class instance with default params for quick and simple usage.All algorithms have some common methods:.distance(*sequences)-- calculate distance between sequences..similarity(*sequences)-- calculate similarity for sequences..maximum(*sequences)-- maximum possible value for distance and similarity. For any sequence:distance + similarity == maximum..normalized_distance(*sequences)-- normalized distance between sequences. The return value is a float between 0 and 1, where 0 means equal, and 1 totally different..normalized_similarity(*sequences)-- normalized similarity for sequences. The return value is a float between 0 and 1, where 0 means totally different, and 1 equal.Most common init arguments:qval-- q-value for split sequences into q-grams. Possible values:1 (default) -- compare sequences by chars.2 or more -- transform sequences to q-grams.None -- split sequences by words.as_set-- for token-based algorithms:True --tandtttis equal.False (default) --tandtttis different.ExamplesFor example,Hamming distance:importtextdistancetextdistance.hamming('test','text')# 1textdistance.hamming.distance('test','text')# 1textdistance.hamming.similarity('test','text')# 3textdistance.hamming.normalized_distance('test','text')# 0.25textdistance.hamming.normalized_similarity('test','text')# 0.75textdistance.Hamming(qval=2).distance('test','text')# 2Any other algorithms have same interface.ArticlesA few articles with examples how to use textdistance in the real world:Guide to Fuzzy Matching with PythonString similarity — the basic know your algorithms guide!Normalized compression distanceExtra librariesFor main algorithms textdistance try to call known external libraries (fastest first) if available (installed in your system) and possible (this implementation can compare this type of sequences).Installtextdistance with extras for this feature.You can disable this by passingexternal=Falseargument on init:importtextdistancehamming=textdistance.Hamming(external=False)hamming('text','testit')# 3Supported libraries:Distancejellyfishpy_stringmatchingpylevLevenshteinpyxDamerauLevenshteinAlgorithms:DamerauLevenshteinHammingJaroJaroWinklerLevenshteinBenchmarksWithout extras installation:algorithmlibrarytimeDamerauLevenshteinrapidfuzz0.00312DamerauLevenshteinjellyfish0.00591DamerauLevenshteinpyxdameraulevenshtein0.03335DamerauLevenshteintextdistance0.83524HammingLevenshtein0.00038Hammingrapidfuzz0.00044Hammingjellyfish0.00091Hammingdistance0.00812Hammingtextdistance0.03531Jarorapidfuzz0.00092Jarojellyfish0.00191Jarotextdistance0.07365JaroWinklerrapidfuzz0.00094JaroWinklerjellyfish0.00195JaroWinklertextdistance0.07501Levenshteinrapidfuzz0.00099LevenshteinLevenshtein0.00122Levenshteinjellyfish0.00254Levenshteinpylev0.15688Levenshteindistance0.28669Levenshteintextdistance0.53902Total: 24 libs.Yeah, so slow. Use TextDistance on production only with extras.Textdistance use benchmark's results for algorithm's optimization and try to call fastest external lib first (if possible).You can run benchmark manually on your system:pipinstalltextdistance[benchmark]python3-mtextdistance.benchmarkTextDistance show benchmarks results table for your system and save libraries priorities intolibraries.jsonfile in TextDistance's folder. This file will be used by textdistance for calling fastest algorithm implementation. Defaultlibraries.jsonalready included in package.Running testsAll you need istask. SeeTaskfile.ymlfor the list of available commands. For example, to run tests including third-party libraries usage, executetask pytest-external:run.ContributingPRs are welcome!Found a bug? Fix it!Want to add more algorithms? Sure! Just make it with the same interface as other algorithms in the lib and add some tests.Can make something faster? Great! Just avoid external dependencies and remember that everything should work not only with strings.Something else that do you think is good? Do it! Just make sure that CI passes and everything from the README is still applicable (interface, features, and so on).Have no time to code? Tell your friends and subscribers abouttextdistance. More users, more contributions, more amazing features.Thank you :heart: