Single IntMap instead of 2 to save some lookups.

Do you have a benchmark to test whether this is an improvement or not?