Write UTF-8 files and remove fragments from URLs.

The UTF-8 bit is a quick fix and I have yet to see how to do that in a
principled way. I don't know what happens when a page uses a different encoding
(how requests handles that.)

Fragments are not sent to the server, so it makes sense that we remove them
(that way we can identify duplicate pages.)
Add support for custom User-Agent strings.

Useful when you want the crawler to appear like e.g.  a specific browser to the
site being crawled.