Updated the dedup api and cleaned up