Meta robots

This commit allows the code to honor the meta attributes for nofollow and noindex. The system is separately configurable to honor nofollow and/or noindex. The default configuration is to honor noindex, but to ignore nofollow.
So I moved the config read into fetcher_test.go. The reason is that I want to be able to run a single file of tests only (so I don't have to wait for the whole suit to run). I invoke as

(cd test && sudo -E go test -v fetcher_test.go helpers.go mocks.go config_test.go) 

But if you forget the config_test.go the tests will fail unexplainably (reason was that the config not being loaded, so blacklisting was preventing some reads). The easiest way to solve is to put an init in fetcher_test: which is the way the code is now. But if there are objections to that, my only goal is to be able to prevent users from having the same problem I did. So an alternative plane might be:

(1) merge test.sh and coverage.sh into a single script
(2) add the ability for the user to call `script/test.sh fetcher` and have the script be smart enough to know which files need to be included in order to run fetcher_test.go

Cleans up our scripts directory, and helps new users adapt to the walker environment. As well as makes it easier for us to test one-file at a time. Where are y'all on this issue?