Remove rounding from caching logic

Not sure if it was really harmful, but we never hit the cache anyways,
so why not simplify the code?