Loop over the much smaller index array when cleaning the LRU cache.

This dramatically decreases the amount of calls to in_array, especially when running tests. I saved 5 seconds locally running 4 test files.