Package: scikit-bloom
Scikit-BloomAn excuse to play with Rust, but also a neat trick for sklearn!This package contains some bloom tricks for text pipelines in scikit-learn. To learn more about this trick, check outthis blogpost.You can install it via:python -m pip install scikit-bloomAnd you can import the components via:fromskbloomimportBloomVectorizer,BloomishVectorizer,SlowBloomVectorizerBloomVectorizer().fit(X).transform(X)BloomishVectorizer().fit(X).transform(X)TheBloomVectorizerwill use rust under the hood for the hashing to construct the bloom representation. TheBloomishVectorizerwill just run theHashingVectorizerfrom scikit-learn multiple times in sequence. TheSlowBloomVectorizeris pretty much the same as theBloomVectizerin terms of features, but is implemented in Python.BenchmarksI ran a quick benchmark, which seems to suggest the approach is pretty speedy.Show me the codeimporttimefromdatasetsimportload_datasetfromskbloomimportBloomVectorizer,BloomishVectorizer,SlowBloomVectorizerfromsklearn.feature_extraction.textimportHashingVectorizerdataset=load_dataset("clinc_oos","plus")texts=dataset['train']['text']*10trials=[BloomVectorizer(n_features=10_000),BloomishVectorizer(n_features=10_000),SlowBloomVectorizer(n_features=10_000),HashingVectorizer(n_features=10_000)]fortrialintrials:tic=time.time()trial.fit_transform(texts)toc=time.time()print(f"{trial.__class_.__name__}:{toc-tic}")In this benchmark we're creating aApproachTime takenDescriptionBloomVectorizer1.562The speedy rust implementationBloomishVectorizer2.111Using sklearn's implementation sequentiallySlowBloomVectorizer5.259A pure python implementationHashingVectorizer0.695Using sklearn's hashing vectorizer to only hash onceYou can also choose to run theBloomVectorizerby just hashing once and it seems to be competative with theHashingVectorizer.Show me the codeimporttimefromdatasetsimportload_datasetfromskbloomimportBloomVectorizer,BloomishVectorizer,SlowBloomVectorizerfromsklearn.feature_extraction.textimportHashingVectorizerdataset=load_dataset("clinc_oos","plus")texts=dataset['train']['text']*10forfeatsin[3000,5000,10000,20000,100_000]:trials=[BloomVectorizer(n_hash=1,n_features=feats),HashingVectorizer(n_features=feats)]fortrialintrials:tic=time.time()trial.fit_transform(texts)toc=time.time()print(f"{feats}:{trial.__class__.__name__}:{toc-tic}")Number of featsBloomVectorizerHashingVectorizer30000.60710.686450000.60920.6947100000.61230.6911200000.61240.69181000000.61080.6938I want to be careful with suggesting that theBloomVectorizeris always faster
because theHashingVectorizercomes with way more features. You can build n-gram representations, just to mention one example, which theBloomVectorizerdoes not do. But it does seem like it is at least competative, which is neat.ImportantIn fairness, while this trick is interesting ... youmightbe fine just using theHashingVectorizerthat just comes with sklearn. This project works, but it was also an excuse for me to try out rust.It's a nice motivating example for me to learn rust, partially because it's a tangible example from a field that I am familiar with. But it's also been a relatively low investment to rewrite an expensive bit of code in rust.DevelopmentThese are mainly some notes for myself.To install all of this locally;python -m pip install maturin 
maturin develop
python -m pip install -e .