Started working with tokenizer
Added Tokenizer project
Changed dictionaries to take key int instead of strings and added a check on the robotstxt file to make sure it is not resolved twice
Changed namespace to WebCrawler.Core instead of .Crawl
Changed dictionaries to take int as key instead of string
Changed namespace to WebCrawler.Core instead of WebCrawler.Crawl
Changed namespace and made getter for IPAddress return an int for the hashed address
Returns null if no robots.txt is found and added a timeout for the request
Renamed solution and webcrawler project
Initial commit of the indexer