mozilla/minion-backend — pull request #39 by yeukhon (2013-06-10 20:32:13+00:00)
Title: Fixed #36. Fixed #11. RobotsPlugin now uses robot-scanner

This PR merges a bunch of branches. We have robot-scanner added to setup.py and we have robots plugin using robot-scanner to scan the content, and we perform additional validation within the ``validator`` method in RobotsPlugin class.