remove use of recycled set in filters eviction

Is it thread-safe? If the cleanup takes very long I think this set might be used by two threads at the same time?