CCoinsKeyHasher::operator() should return size_t

It currently returns uint64_t, which on older boost (at least 1.46) causes failures on 32-bit systems. This problem was introduced in bc42503.

Fixes #4634.