improve memoize with size limit

- previous implementation was dropping complete
  hash-table whenever max size was reached

- new implementation simply replaces old values
  in a fixed size hash-table when hashes collide

- use cuckoo hashing for better space usage and to
  reduce collisions of frequently used values
Merge pull request #2591 from MartinNowak/memoize

improve memoize with size limit