add comment for CrawlDecisionMaker and MultiThreadManager
Add class, methods for wcrawler.robotstxt
Merge branch 'first' of https://github.com/swdgroup1/crawlergroup1.git into first