Rewrote spider to try to process scraped data before requesting next pages.