Renamed IPrioritizer and SimplePrioritizer to match the teminology used in the literature
mend
mend
updated robots.txt parser and added functionality to check whether an url is legal to visit
Moved IURLFrontier and SimpleURLFrontier to a seperate folder
Rearranged some logic in the crawler
Implemented politeness and fixed a bug where Sitemaps would not be initialzed in RobotsTxt.cs
moved seeds.txt to console project