Scanner is now capable of scanning robots.txt

In order to actually read Disallow field, we have to combine user_agent
and user_agent_value, disallow and disallow_value into a single regex.
This may be a little more efficient in terms of scanning because we are
doing fewer callbacks and scanning, and regex matching is more narrow.
On the other hand, the regex is not most optimal. We have to consult
with Python regex documentation.

Next issue is comment. The scanner should output a list of tokens and we
should not include comment fields. This is trival. We also need to look
at possible adoption of generators: are they useful? We need to run some
real stress/performance tests with a lot of robots.

The robots.txt included contains entries from
http://www.robotstxt.org/robotstxt.html,

TODO list:

1. output tokens in a list/tuple excluding comments
2. try performance tests with a lot of robots
3. validate and invalidate robots.txt
4. make use of generators?
5. make our regex more optimized if possible
Added scan() and returns a tuple of tokens.