Package: randomhash
PythonrandomhashpackageA simple, time-tested, family of random hash functions in Python, based on CRC32
and xxHash, affine transformations, and the Mersenne Twister.This is a companion library tothe identical Java version.Installation and usageThe library is available on PyPI, and can be installed through normal means:$pipinstallrandomhashOnce installed, it can be called either by instantiating a family of random
hash functions, or using the default instantiated functions:importrandomhash# Create a family of random hash functions, with 10 hash functionsrfh=randomhash.RandomHashFamily(count=10)print(rfh.hashes("hello"))# will compute the ten hashes for "hello"# Use the default instantiated functionsprint(randomhash.hashes("hello",count=10))FeaturesThis introduces a family of hash functions that can be used to implement probabilistic
algorithms such as HyperLogLog. It is based onaffine transformations of either the
CRC32 hash functions, which have been empirically shown to provide good performance
(for consistency with other versions of this library, such as the Java version), orthe more complex xxHash hash functionsthat are
made available throughthexxhashPython bindings.
The pseudo-random numbers are drawn according tothe standard Python implementationof theMersenne Twister.Some historyIn 1983, G. N. N. Martin and Philippe Flajolet introduced the algorithm known
asProbabilistic Counting,
designed to provide an extremely accurate and efficient
estimate of the number of unique words from a document that may contain repetitions.
This was an incredibly important algorithm, which introduced arevolutionary ideaat the time:The only assumption made is that records can be hashed in a suitably pseudo-uniform
manner. This does not however appear to be a severe limitation since empirical
studies on large industrial files [5] reveal thatcarefulimplementations of
standard hashing techniques do achieve practically uniformity of hashed values.The idea is that hash functions can "transform" data into pseudo-random variables.
Then a text can be treated as a sequence of random variables drawn from a uniform
distribution, where a given word will always occur as the same random value (i.e.,a b c a a b ccould be hashed as.00889 .31423 .70893 .00889 .00889 .31423 .70893with
every occurrence ofahashing to the same value). While this sounds strange,
empirical evidence suggests it is true enough in practice, and eventuallysome
theoretical basishas come to support the practice.The originalProbabilistic Counting(1983) algorithm gave way toLogLog(2004),
and then eventuallyHyperLogLog(2007), one of the most famous algorithms in the
world as described inthis article. These algorithms
and others all used the same idea of hashing inputs to treat them as random variables,
and proved remarkably efficient and accurate.But as highlighted in the above passage, it is important to becareful.Hash functions in practiceIn practice, it is easy to use poor quality hash functions, or to use cryptographic
functions which will significantly slow down the speed (and relevance) of the
probabilistic estimates. However, on most data, some the cyclic polynomial checksums
(such as Adler32 or CRC32) provide good results---as do efficient, general-purpose
non-cryptographic hash functions such as xxHash.