Meta robots

This commit allows the code to honor the meta attributes for nofollow and noindex. The system is separately configurable to honor nofollow and/or noindex. The default configuration is to honor noindex, but to ignore nofollow.
By the way I think it'll make more sense if we differentiate what walker will get links from and what we will pass to the handler. I could imagine, for example, people wanting to crawl a site but only have their handler called for PDF files.

I'm going to do some to break this code down soon anyway so we can leave it as is for this merge. If that sounds okay to you I'll do it next.