Only clean up the cache when we have doubled its size.