refactored for calling from scraper