removed test condition, scraper takes about 1 hour to run