Added basic unduplicate handling