* Added support for robotstxt module.