Remove unnecessary allocations in ISet