Package: simplefuzzyset
Forked fromhttps://github.com/axiak/fuzzysetRemoved levenshtein supportNo dependencies on installfuzzyset is a data structure that performs something akin to fulltext search
against data to determine likely mispellings and approximate string matching.UsageThe usage is simple. Just add a string to the set, and ask for it later
by using either.getor[]:>>> a = fuzzyset.FuzzySet()
>>> a.add("michael axiak")
>>> a.get("micael asiak")
[(0.8461538461538461, u'michael axiak')]The result will be a list of(score, mached_value)tuples.
The score is between 0 and 1, with 1 being a perfect match.For roughly 15% performance increase, there is also a Cython-implemented
version calledcfuzzyset. So you can write the following, akin tocStringIOandcPickle:try:
    from cfuzzyset import cFuzzySet as FuzzySet
except ImportError:
    from fuzzyset import FuzzySetConstruction Argumentsiterable: An iterable that yields strings to initialize the data structure withgram_size_lower: The lower bound of gram sizes to use, inclusive (see Theory of operation). Default: 2gram_size_upper: The upper bound of gram sizes to use, inclusive (see Theory of operation). Default: 3use_levenshtein: Whether or not to use the levenshtein distance to determine the match scoring. Default: TrueTheory of operationAdding to the data structureFirst let’s look at adding a string, ‘michaelich’ to an empty set. We first break apart the string into n-grams (strings of length
n). So trigrams of ‘michaelich’ would look like:'-mi'
'mic'
'ich'
'cha'
'hae'
'ael'
'eli'
'lic'
'ich'
'ch-'Note that fuzzyset will first normalize the string by removing non word characters except for spaces and commas and force
everything to be lowercase.Next the fuzzyset essentially creates a reverse index on those grams. Maintaining a dictionary that says:'mic' -> (1, 0)
'ich' -> (2, 0)
...And there’s a list that looks like:[(3.31, 'michaelich')]Note that we maintain this reverse index forallgrams fromgram_size_lowertogram_size_upperin the constructor.
This becomes important in a second.RetrievingTo search the data structure, we take the n-grams of the query string and perform a reverse index look up. To illustrate,
let’s consider looking up'michael'in our fictitious set containing'michaelich'where thegram_size_upperandgram_size_lowerparameters are default (3 and 2 respectively).We begin by considering first all trigrams (the value ofgram_size_upper). Those grams are:'-mi'
'mic'
'ich'
'cha'
'el-'Then we create a list of any element in the set that hasat least oneoccurrence of a trigram listed above. Note that
this is just a dictionary lookup 5 times. For each of these matched elements, we compute thecosine similaritybetween
each element and the query string. We then sort to get the most similar matched elements.Ifuse_levenshteinis false, then we return all top matched elements with the same cosine similarity.Ifuse_levenshteinis true, then we truncate the possible search space to 50, compute a score based on the levenshtein
distance (so that we handle transpositions), and return based on that.In the event that none of the trigrams matched, we try the whole thing again with bigrams (note though that if there are no matches,
the failure to match will be quick). Bigram searching will always be slower because there will be a much larger set to order.Installeasy_install fuzzysetLicenseBSDAuthorMike Axiak <mike@axiak.net>