Hash#except is expensive and this is a hotspot, so use a dup + delete