checks for duplicates on successive scrapes