Package: nmd
N-gram Mover's DistanceA string similarity measure based on Earth Mover's DistanceSeengram-movers-distancefor codeWhy another string matching algorithm?Edit distance really wasn't cutting it when I needed to look up a dictionary for a misspelled wordWith an edit distance of 1 or 2, the results are not useful since the target word isn't foundWith a distance >=5, the results are meaningless since it contains half the dictionarySame goes for Damerau-LevenshteinAlso, edit distance is pretty slow when looking up long words in a large dictionaryEven after building a finite state automaton or using a trie to optimize lookupNMD was designed with indexing in mindA simpler index could be used for Jaccard or cosine similarity over ngramsEMD (and hence NMD) can be optimized to run really fast with some constraintsValues are 1-dimensional scalarsValues are always quantizedUsagengram_movers_distance()string distance metric, use this to compare two stringsfromnmdimportngram_movers_distance# n-gram mover's distanceprint(ngram_movers_distance(f'hello',f'yellow'))# similarity (inverted distance)print(ngram_movers_distance(f'hello',f'yellow',invert=True))# distance, normalized to the range 0 to 1 (inclusive of 0 and 1)print(ngram_movers_distance(f'hello',f'yellow',normalize=True))# similarity, normalized to the range 0 to 1 (inclusive of 0 and 1)print(ngram_movers_distance(f'hello',f'yellow',invert=True,normalize=True))WordListuse this for dictionary lookups of wordsfromnmdimportWordList# get words from a text filewithopen(f'dictionary.txt',encoding=f'utf8')asf:words=set(f.read().split())# index wordsword_list=WordList((2,4),filter_n=0)# combined 2- and 4-grams seem to work bestforwordinwords:word_list.add_word(word)# lookup a wordprint(word_list.lookup(f'asalamalaikum'))# -> 'assalamualaikum'print(word_list.lookup(f'walaikumalasam'))# -> 'waalaikumsalam'bow_ngram_movers_distance()WARNING: requiresscipy.optimize, so it's not available by default in thenmdnamespaceuse this to compare sequences of tokens (not necessarily unique)note that this does not merge or split words, so if you're matching["pineapple"]and["pine", "apple"]the
similarity will be low. consider just using nmd in this case.fromnmd.nmd_bowimportbow_ngram_movers_distancetext_1=f'Clementi Sports Hub'text_2=f'sport hubs clemmeti'print(bow_ngram_movers_distance(bag_of_words_1=text_1.casefold().split(),bag_of_words_2=text_2.casefold().split(),invert=True,# invert: return similarity instead of distancenormalize=True,# return a score between 0 and 1))todotodo: trythis paper's algowhich referencedthis paperuse less bizarre test stringsnote where the algorithm breaks downmatching long strings with many n-gramsmatching strings with significantly different lengthsrename nmd_bow because it isn't really a bag-of-words, it's a token sequenceconsider areal_quick_ratio-like optimization, or maybe calculate length bounds?needs a cutoff to actually speed up though, makes a huge difference for diffliba sufficiently low cutoff is not unreasonable, although the default of 0.6 might be a little high for nmdthat said the builtin diff performs pretty badly at low similarities, so 0.6 is reasonable for themdefreal_quick_ratio(self):"""Return an upper bound on ratio() very quickly.This isn't defined beyond that it is an upper bound on .ratio(), andis faster to compute than either .ratio() or .quick_ratio()."""la,lb=len(self.a),len(self.b)# can't have more matches than the number of elements in the shorter sequencematches,length=min(la,lb),la+lbiflength:return2.0*matches/lengthreturn1.0create a better string container for the index, more like asetadd(word: str)remove(word: str)clear()__contains__(word: str)__iter__()better lookupadd a min_similarity filter (float, based on normalized distance)lookup(word: str, min_similarity: float = 0, filter: bool = True)try__contains__firsttry levenshtein automaton (distance=1) second?sort by nmd, since most likely there will only be a few resultsbut how to get multiple results?still need to run full search?or maybe just return top 1 result?prefix lookuplook for all strings that are approximately prefixedlike existing index but not normalized and ignoring unmatched ngrams from targetPublishing (notes for myself)initpip install flitflit initmake surenmd/__init__.pycontains a docstring and versionpublish / updateincrement__version__innmd/__init__.pyflit publish