dedupe as you go