testing each url now and outputting to output.txt
need utf8 to catch foreign chars
flush output so i can tee the output
better agent to avoid HTTP/406
cookies and referrer to look more like a browser
forgot to put a referrer for the HEAD req too
Merge branch 'regexp_parens_in_url' of github.com:borgified/url_checker into regexp_parens_in_url