Added a very basic validator to robots plugin.

This basic validator will address some of the concerns in #11 and in
ticket #36. We are rejecting all requests returning non-200 response. If
the header does not return with `text/plain` we assume it is also an
invalid robots.txt.

We will have a difficult time using robotparser straight from Python's
standard library. The module does not return the content of robots.txt.
Subclassing the module may be trick as implementing based on its
"already-parse" content is useless. I also want to give a try to
implement a real robots.txt parser according to Google's document
outlined in [1]. I believe the first version could just follow the
recommendation based on [2] and [3].

[1]:
https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt

[2]: http://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1

[3]: http://www.robotstxt.org/orig.html