Loading a frontier from a non-existing file returns an empty frontier.
Included an extention that simplifies binary search in lists.
The Frontier uses a sorted list for 'allIKnow' using binary search.
Shingle-splitting is no longer done using Skip and Take, this increases speed.
Simplified exclusions, such that only relevat user agents are parsed.
Removed the Load/Save feature from frontier.
Improved the output provided by the console.
Added a threaded frontier that uses a dedicated thread to test if URLs are known to the frontier and if they can be visited by a robot.
Renamed StemmerInterface -> IStemmer.
Loading will only stop when the load.count reaches 30 - it is not tested if the frontier is empty.
Renamed the project/namespace Crawler -> WebCrawler.
Added Crawler class stub for encapsulating the crawler details.
ThreadedFrontier.Next returns 'null' if no URLs are enqueued.
Program uses the new definition of the Next method and crawls until null is retrieved.
Copied the GetLinks method from Program to the Crawler.
Copied the main part of Program into the Crawler class.
Fixed a few bugs with the ThreadedFrontier thread.
Removed all console-output (except when url load is complete) from the Crawler class.