alphagov/govuk_crawler_worker — pull request #40 by dcarley (2014-07-30 15:25:25+00:00)
Title: [Fixes #75962332] Relax URL path sanitisation

Just use path.Clean() to remove/resolve traversals, which will prevent the
file from being written outside of our root. The kennygrant/sanitize package
was doing too much as described in the story and covered by the additional
test cases. We don't strictly need these now that we've removed that package
but I think it might make sense to prevent regressions if we revisit this in
the future.

NB: About non-latin characters. It seems that the old mirror wrote these
filenames as URL encoded. However some testing shows that Nginx on the
mirror machines won't serve these as desired:

root@mirror0:/srv/mirror_data/www-origin# echo one > 如何在香港申請英國簽證.html
root@mirror0:/srv/mirror_data/www-origin# echo two > %E5%A6%82%E4%BD%95%E5%9C%A8%E9%A6%99%E6%B8%AF%E7%94%B3%E8%AB%8B%E8%8B%B1%E5%9C%8B%E7%B0%BD%E8%AD%89.html
root@mirror0:/srv/mirror_data/www-origin# curl -kH "Host: www-origin.mirror.foo" https://localhost/如何在香港申請英國簽證
one
root@mirror0:/srv/mirror_data/www-origin# curl -kH "Host: www-origin.mirror.foo" https://localhost/%E5%A6%82%E4%BD%95%E5%9C%A8%E9%A6%99%E6%B8%AF%E7%94%B3%E8%AB%8B%E8%8B%B1%E5%9C%8B%E7%B0%BD%E8%AD%89one

Our default behaviour of not URL encoding (and now not stripping) these is
correct.