Fixed #36. Fixed #11. RobotsPlugin now uses robot-scanner

This PR merges a bunch of branches. We have robot-scanner added to setup.py and we have robots plugin using robot-scanner to scan the content, and we perform additional validation within the ``validator`` method in RobotsPlugin class.