axiak/pybloomfiltermmap — Issue #16: Item hashes differ between unicode and str, and 32 / 64 bits
Author: asl2
Created: 2011-12-19 22:44:13+00:00
State: open

I noticed that unlike the built-in containers, pybloomfiltermmap differentiates between unicode and str, eg

>>> from pybloomfilter import BloomFilter
>>> filter = BloomFilter(100000, 0.01, '/tmp/bloomfilter')
>>> filter.add(u'foo')
False
>>> 'foo' in filter
False
>>> u'foo' in filter
True
>>> 'foo' in set([u'foo'])
True

(The only documentation I can find for this is [PEP-100] [1], which makes no guarantees about hashes of unicode containing non-ascii characters.)
[1]:http://www.python.org/dev/peps/pep-0100/ "PEP-100"

I also noticed that the default python hash on 32-bit python is the low 32 bits of the hash on 64-bit python. For example, on 32-bit:

>>> hex(hash("foo"))
'-0x2c217945'

vs 64-bit:

>>> hex(hash("foo"))
'-0x39f8634c2c217945'

This caused me some head-scratching: my workaround was to encode unicode strings via utf-8, so they'd go through the string hashing instead of python hashing, and it works splendidly now.

At the next file-format incompatibility, it might be nice to hash unicode strings via PyString_AsStringAndSize, and mask off the top 32 bits on 64-bit systems.