Don't check for robots.txt directives in local html files.
Don't check non-connected URLs.
Add missing log keyword arg.
Print SSL cipher info for https URLs.