Added support for parsing crawl-delay
Merge pull request #5 from troelskn/feature-support-crawl-delay

Added support for parsing crawl-delay - thanks to https://github.com/troelskn