Default port is now 6464, switched to generic user agent
Removed the option to disable obeying directives in robots.txt