Shallow copy of set and the remove is faster than set intersection.