Cache memory dispatch lookups

Removes a layer of lookup for memory access when we get a cache hit in
our home-rolled caching mechanism.