Moved "do we put this new archive in the darchive?" logic to the
warcprox task. This will help us archive the robots.txt file and
provides a better way for us to check on the noarchive tag in
the markup.

TODO: We'll want to refactor this robots stuff a bit. We should
clearly be getting the title of the page in the same way we look
for the noarchive tag in the markup. This is part of the larger
"run as much through phantomjs as possible" task.

fixes #461
fixes #469
Merge branch 'develop' of github.com:harvard-lil/perma into develop