Getting rid of 2 new HashSets by iterating over the non-removable values of the cache rather than getting a copy (of a copy)