improve speed by checking duplicates before pulling