add scraper validation and duplicate checking