finished deduplicate, fixed scraping error