buildWith' already calls removeNonexistent, so don't do it twice

From a quick look, it seems like removeNonexistent should be idempotent. So let's only do it once, save the GC a little effort.