Package: neofuzz
NeofuzzBlazing fast, lightweight and customizable fuzzy and semantic text search in Python.Introduction (Documentation)Neofuzz is a fuzzy search library based on vectorization and approximate nearest neighbour
search techniques.Why is Neofuzz fast?Most fuzzy search libraries rely on optimizing the hell out of the same couple of fuzzy search algorithms (Hamming distance, Levenshtein distance). Sometimes unfortunately due to the complexity of these algorithms, no amount of optimization will get you the speed, that you want.Neofuzz makes the realization, that you can’t go above a certain speed limit by relying on traditional algorithms, and uses text vectorization and approximate nearest neighbour search in the vector space to speed up this process.When it comes to the dilemma of speed versus accuracy, Neofuzz goes full-on speed.When should I choose Neofuzz?You need to do repeated searches in the same corpus.Levenshtein and Hamming distance is simply not fast enough.You are willing to sacrifice the quality of the results for speed.You don’t mind that the up-front computation to index a corpus might take time.You have very long strings, where other methods would be impractical.You want to rely on semantic content.You need a drop-in replacement for TheFuzz.When should I NOT choose Neofuzz?The corpus changes all the time, or you only want to do one search in a corpus. (It might still give speed-up in that case though.)You value the quality of the results over speed.You don’t mind slower searches in favor of no indexing.You have a small corpus with short strings.UsageYou can install Neofuzz from PyPI:pipinstallneofuzzIf you want a plug-and play experience you can create a generally good quick and dirty
process with thechar_ngram_process()process.fromneofuzzimportchar_ngram_process# We create a process that takes character 1 to 5-grams as features for# vectorization and uses a tf-idf weighting scheme.# We will use cosine distance for the nearest neighbour search.process=char_ngram_process(ngram_range=(1,5),metrics="cosine",tf_idf=True)# We index the options that we are going to search inprocess.index(options)# Then we can extract the ten most similar items the same way as in# thefuzzprocess.extract("fuzz",limit=10)---------------------------------[('fuzzer',67),('Januzzi',30),('Figliuzzi',25),('Fun',20),('Erika_Petruzzi',20),('zu',20),('Zo',18),('blog_BuzzMachine',18),('LW_Todd_Bertuzzi',18),('OFU',17)]Custom ProcessesYou can customize Neofuzz’s behaviour by making a custom process.
Under the hood every Neofuzz Process relies on the same two components:A vectorizer, which turns texts into a vectorized form, and can be fully customized.Approximate Nearest Neighbour search, which indexes the vector space and can find neighbours of a given vector very quickly. This component is fixed to be PyNNDescent, but all of its parameters are exposed in the API, so its behaviour can also be altered at will.Words as FeaturesIf you’re more interested in the words/semantic content of the text you can also use them as features. This can be very useful especially with longer texts, such as literary works.fromneofuzzimportProcessfromsklearn.feature_extraction.textimportTfidfVectorizer# Vectorization with words is the default in sklearn.vectorizer=TfidfVectorizer()# We use cosine distance because it's waay better for high-dimentional spaces.process=Process(vectorizer,metric="cosine")Dimentionality ReductionYou might find that the speed of your fuzzy search process is not sufficient. In this case it might be desirable to reduce the dimentionality of the produced vectors with some matrix decomposition method or topic model.Here for example I use NMF (excellent topic model and incredibly fast one too) too speed up my fuzzy search pipeline.fromneofuzzimportProcessfromsklearn.feature_extraction.textimportTfidfVectorizerfromsklearn.decompositionimportNMFfromsklear.pipelineimportmake_pipeline# Vectorization with tokens againvectorizer=TfidfVectorizer()# Dimentionality reduction method to 20 dimentionsnmf=NMF(n_components=20)# Create a pipeline of the twopipeline=make_pipeline(vectorizer,nmf)process=Process(pipeline,metric="cosine")Semantic Search/Large Language ModelsWith Neofuzz you can easily use semantic embeddings to your advantage, and can use both attention-based language models (Bert), just simple neural word or document embeddings (Word2Vec, Doc2Vec, FastText, etc.) or even OpenAI’s LLMs.We recommend you try embetter, which has a lot of built-in sklearn compatible vectorizers.pipinstallembetterfromembetter.textimportSentenceEncoderfromneofuzzimportProcess# Here we will use a pretrained Bert sentence encoder as vectorizervectorizer=SentenceEncoder("all-distilroberta-v1")# Then we make a process with the language modelprocess=Process(vectorizer,metric="cosine")# Remember that the options STILL have to be indexed even though you have a pretrained vectorizerprocess.index(options)