CCoinsKeyHasher::operator() should return size_t

It currently returns uint64_t, which on older boost (at least 1.46) causes
test failures on 32-bit systems.

This problem was introduced in bc42503.

Fixes #4634.