Working and tested robots.txt parser

* Works only on simple cases of Disallow
Adds case insensitivity to user agent matching