Removed an optimization that caused problems on 32 bit builds

The speed up was only about 30% anyway and irrelevant after the cache is created.