Factor out the scraping from the middleware