Implemented more robust robots.txt policy. Altered return for read_html; no longer returns host information. Implemented sleep policy.
Added domain constructor parameter. Fixed robots.txt parsing. Added support for domain regex. Fixed bug in resolving relative paths.