Package: pyprobables
pyprobablesis a pure-python library for probabilistic data structures.
The goal is to provide the developer with a pure-python implementation of
common probabilistic data-structures to use in their work.To achieve better raw performance, it is recommended supplying an alternative
hashing algorithm that has been compiled in C. This could include using the
md5 and sha512 algorithms provided or installing a third party package and
writing your own hashing strategy. Some options include the murmur hashmmh3or those from thepyhashlibrary. Each data object inpyprobablesmakes it easy to pass in a custom hashing function.Read more about how to useSupplying a pre-defined, alternative hashing strategiesorDefining hashing function using the provided decorators.InstallationPip Installation:$ pip install pyprobablesTo install from source:To installpyprobables, simply clone therepository on GitHub, then run from the folder:$ python setup.py installpyprobablessupports python 3.6 - 3.11+Forpython 2.7support, installrelease 0.3.2$ pip install pyprobables==0.3.2API DocumentationThe documentation of is hosted onreadthedocs.ioYou can build the documentation locally by running:$ pip install sphinx
$ cd docs/
$ make htmlAutomated TestsTo run automated tests, one must simply run the following command from the
downloaded folder:$ python setup.py testQuickstartImport pyprobables and setup a Bloom FilterfromprobablesimportBloomFilterblm=BloomFilter(est_elements=1000,false_positive_rate=0.05)blm.add('google.com')blm.check('facebook.com')# should return Falseblm.check('google.com')# should return TrueImport pyprobables and setup a Count-Min SketchfromprobablesimportCountMinSketchcms=CountMinSketch(width=1000,depth=5)cms.add('google.com')# should return 1cms.add('facebook.com',25)# insert 25 at once; should return 25Import pyprobables and setup a Cuckoo FilterfromprobablesimportCuckooFiltercko=CuckooFilter(capacity=100,max_swaps=10)cko.add('google.com')cko.check('facebook.com')# should return Falsecko.check('google.com')# should return TrueImport pyprobables and setup a Quotient FilterfromprobablesimportQuotientFilterqf=QuotientFilter(quotient=24)qf.add('google.com')qf.check('facebook.com')# should return Falseqf.check('google.com')# should return TrueSupplying a pre-defined, alternative hashing strategiesfromprobablesimportBloomFilterfromprobables.hashesimportdefault_sha256blm=BloomFilter(est_elements=1000,false_positive_rate=0.05,hash_function=default_sha256)blm.add('google.com')blm.check('facebook.com')# should return Falseblm.check('google.com')# should return TrueDefining hashing function using the provided decoratorsimportmmh3# murmur hash 3 implementation (pip install mmh3)fromprobables.hashesimporthash_with_depth_bytesfromprobablesimportBloomFilter@hash_with_depth_bytesdefmy_hash(key,depth):returnmmh3.hash_bytes(key,seed=depth)blm=BloomFilter(est_elements=1000,false_positive_rate=0.05,hash_function=my_hash)importhashlibfromprobables.hashesimporthash_with_depth_intfromprobables.constantsimportUINT64_T_MAXfromprobablesimportBloomFilter@hash_with_depth_intdefmy_hash(key,seed=0,encoding="utf-8"):max64mod=UINT64_T_MAX+1val=int(hashlib.sha512(key.encode(encoding)).hexdigest(),16)val+=seed# not a good example, but uses the seed valuereturnval%max64modblm=BloomFilter(est_elements=1000,false_positive_rate=0.05,hash_function=my_hash)See theAPI documentationfor other data structures available and thequickstart pagefor more examples!ChangelogPlease see thechangelogfor a list
of all changes.Backward Compatible ChangesIf you are using previously exported probablistic data structures (v0.4.1 or below)
and used the default hashing strategy, you will want to use the following code
to mimic the original default hashing algorithm.fromprobablesimportBloomFilterfromprobables.hashesimporthash_with_depth_int@hash_with_depth_intdefold_fnv1a(key,depth=1):returntmp_fnv_1a(key)deftmp_fnv_1a(key):max64mod=UINT64_T_MAX+1hval=14695981039346656073fnv_64_prime=1099511628211tmp=map(ord,key)fort_strintmp:hval^=t_strhval*=fnv_64_primehval%=max64modreturnhvalblm=BloomFilter(filpath="old-file-path.blm",hash_function=old_fnv1a)