FIX issue 35: DefaultRobotsTxtProvider fails parsing robots.txt

* Add handling for commented lines.
* Add handling for partly commented lines.
* Skip rules which allows everything (Allow: /).
* Read all rules which should result in a filter.

The regex question mark on line 222 in DefaultRobotsTxtProvider.java was needed in order to restrict the crawler from crawling the restricted path (The path /spidertrap/ should not be crawled when Disallow: /spidertrap/ is given as a rule).