Lots of bug fixes.

Many small problems, fixed them all and now we are actually crawling!
Rate limiting seems to be working, might be able to tweak the extra time
padding that was added to ensure that we are under the limit.