Fixed #7. Add support for crawl-delay. Accepts int, decimal and frac.
Fixed #6. Add request-rate support.
Added support for visit-time directive. Fixed #8
Ignore swap files.