configurable API crawler, for scraping to static files