fixed bugs in crawler and scraper to eliminate duplicates