core scrape logic