linkcrawler refactor faster tests with threading and no duplicates

added check status codes logic to linkcrawler
remove duplicate urls
thread the checking
simplify code needed in tests