Manually Merged 1.2 branch (still under construction) into 1.3
Sync up with some of Ilyas changes. Added IsRespectRobotsNoFollowEnabled and IsRespectAnchorRelNoFollowEnabled. Left out (DeepCrawlScheduler and Http async client, etc..)
Create README.md
112: Added to auto detect/handle response encodings, no tests
112: Added auto encoding plus tests, however not used by crawler yet
112 & 113: Added auto encoding and CrawledPage.Content.Bytes which should allow data to be writtent to file stream without corruption.
Issue 118: Fixed issue with robots always disallowing external links
Issue 119: Fixed bug with maxpagestocrawl since we now check ShouldCrawlPage before scheduling, not after scheduling. No need to stop the crawl when maxpages has been reached anymore since the check is done on the main thread.
Added stubs for new Scheduler that takes repos for crawled and tocrawl
Added tests to the new generic Scheduler class
Added IsDeepCrawlEnabled, not used by crawled yet
Issue 92: Created ShallowWebCrawler and DeepWebCrawler that are optimized for either deep crawls (store on disk) or shallow crawler (store in memory)
Revert "Added IsDeepCrawlEnabled, not used by crawled yet"

This reverts commit 9e2a11c1250b0106ccc39cdf4f65e54449450dd5.
Added link to docs
Added link to docs
Added link to docs
Added link to docs
Removed a line to allow the solution to be opened in monodevelop. Hopefully it also solves a weird issue discribed here https://groups.google.com/forum/#!searchin/abot-web-crawler/crashing/abot-web-crawler/gEy6y3_GT6U/rpUR8mA5HnoJ
Merge branch 'master' of github.com:sjdirect/abot into 1.2.3

Conflicts:
	Abot.Tests.Unit/Crawler/WebCrawlerTest.cs
Merge branch 'master' of github.com:sjdirect/abot into 1.2.3

Conflicts:
	README.md