t1gor/Robots.txt-Parser-Class — pull request #7 by jlicht (2014-07-08 16:00:17+00:00)
Title: Empty "Disallow:" directives should not mean "disallow all"

A directive of the form:

Disallow:

means that all urls are allowed, not that they are all denied.

See the first entry here: http://en.wikipedia.org/wiki/Robots.txt#Examples