Forgot I was only scraping the first 5 posts on each index page.
Use lxml instead.
Cleaned up initial task queueing.