Fixes WARCRecorder not saving HTTP headers.
URLInfo normalizes default ports.
Implements handling of robots.txt