Added UserAgent to HttpClient
Robots implementation - changing to rules list
Robots implemented in Scraper
Moved rules into dictionary
Tidied up AsRegexRule
Introduced Rule class to Robots
Removed UserAgentName from Scraper
Handle null robots.txt