taganaka/polipus — pull request #30 by taganaka (2014-06-01 19:51:52+00:00)
Title: Support for robots.txt

If enabled it will obey to the robots.txt directive.
If the UserAgent used is not whitelisted, polipus will refuse to follow urls.

Fancy robots.txt directives such as delay are not supported