That works, but splitting the conditional is probably less performant than asking the Set to remove an element that it doesn't contain.  Either way, it's a trivial difference.