linkcrawler refactor:
added check status codes logic to linkcrawler
remove duplicate urls
thread the checking
simplify code needed in tests
updated based on comments so Assert.true could be used
Merge pull request #53 from brann/linkcrawler-threading

linkcrawler refactor faster tests with threading and no duplicates