break the dedup script into 2 parts