Added support for parsing crawl-delay

Sorry for the white space changes. My editor automatically strips end-of-line spaces.