check for duplicates while fetching from data providers