fixes #19 HasRobotsNoFollow does not support "none"
Merge branch 'master' of github.com:sjdirect/abot
fixes #19 slight refactor of the previous checkin to allow base class to handle shared logic
fixes issue # 22 by wiring up PageToCrawl.IsRetry
Merge branch 'master' of github.com:sjdirect/abot into 2.0.0
Added a check for isretry to this engine since it was added to master while this file was new in another branch.