Working on #36 adding test and validator to robots plugin.

Currently robots plugin does not actually verify the
existence of robots.txt. If the target url is 200-reacable, plugin
will return success. This is address by ticket #36.

Ticket #11 addresses the lack of validator in plugin. We should verify
the robots.txt is valid and is confronting to the standard.

We probably will adaopt the recommendations from the official
Robots.org, in addition to w3c.

[1]: http://www.robotstxt.org/orig.html

[2]: http://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1
Added a very basic validator to robots plugin.

This basic validator will address some of the concerns in #11 and in
ticket #36. We are rejecting all requests returning non-200 response. If
the header does not return with `text/plain` we assume it is also an
invalid robots.txt.

We will have a difficult time using robotparser straight from Python's
standard library. The module does not return the content of robots.txt.
Subclassing the module may be trick as implementing based on its
"already-parse" content is useless. I also want to give a try to
implement a real robots.txt parser according to Google's document
outlined in [1]. I believe the first version could just follow the
recommendation based on [2] and [3].

[1]:
https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt

[2]: http://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1

[3]: http://www.robotstxt.org/orig.html
Added functional tests for ServerDetailsPlugin.
Fixed #31 test and fixed #32 header name bug for X-XSS-Protection.

In this commit, we have added a functional test for
XXSSProtectionPlugin. Furthermore, like #27, we have to read the
lower-case header name for X-XSS-Protection.

We should actually address this issue either with curly.py or
individually with each plugin. See issue #33.
Instruct travis to create dist for minion and use pip to install minion.
Instruct travis to install stunnel.
use sudo to kill on travis.
Instruct travis to start mongodb.

We were getting "ConnectionFailure: could not connect to
127.0.0.1:27017: [Errno 111] Connection refused" error in Travis.
Instruct travis to start minions.
Tells travis to create a basic plan.
Fixed #38: robots-scanner is added to setup.py
Fixed #36. Fixed #11. RobotsPlugins now use robots-scanner validator.

The robots-scanner has a very simple scanner to scan the content of
robots.txt. The overall validation in RobotsPlugins is written in the
commit (see plugins/basics.py).
Use robots-scanner 0.1.1.

robots-scanner has been updated to remove print statements from the its
source code.
Removed test_robots_validator.py.

This is unused and the tests for the actual robots-scanner tests are
within robots-scanner source code. We should instead forcus on writing
unittests for validator method.
Merge pull request #39 from yeukhon/robot-plugin-ftests

Fixed #36. Fixed #11. RobotsPlugin now uses robot-scanner