Implemented noindex,noforward handling. NO configuration yet.
Added tests for noindex/nofollow.
Merge branch 'master' into MetaRobots
Added configuration variables for meta-nofollow/noindex handling
Fixed testing bug: moved readConfig earlier in method.
Updated documentation of parseHtml
Changed readConfig -> init in test/fetcher_test.go.
Changed the name of shouldStoreProtocol -> shouldStoreParsedLink
Merge pull request #55 from iParadigms/MetaRobots

Meta robots