Keep a cache of indexes to filter out the duplicates