harvard-lil/perma — pull request #492 by jcushman (2014-04-06 14:30:29+00:00)
Title: Selenium, Heroku, file paths, etc.

This started with switching to Selenium to control our headless browser, which in turn implicated our file storage paths, which in turn implicated Heroku and PyWB. So it kind of turned into a monster update -- oops. I'm just going to throw it all at you rather than try to detangle.

Larger changes:

* Use Selenium to drive PhantomJS from Python. This gives us granular control over each step of indexing, so we can use intelligent fallbacks if onLoad fails to fire, monitor post-load requests, etc. This also lets us use the PhantomJS to retrieve meta tags, title etc., so should be more robust for that stuff.
* Update our templates and code to consistently use STATIC_ROOT and STATIC_URL to refer to our static files, and MEDIA_ROOT and MEDIA_URL to refer to our user-generated files.
- NOTE: When updating, be sure those are set to something useful. `settings_dev.py` contains settings for the Vagrant dev setup, which currently keeps everything in services/django/. (It also keeps user-generated logs in services/logs/ and user-generated data in services/mysql/ . Possibly that could all go somewhere else like media/ or local/ ?)
* Heroku integration: Perma can now be pushed to Heroku with `fab heroku_push`. Heroku config files are in services/heroku/. This is still a work in progress (it works in read-only mode, but isn't totally set up to write or index warcs yet).
* PyWB -- this includes Ilya's PyWB pull request, since I wanted to test it with my new code. Should be fine to pull just this one.
- NOTE: Since the pywb package now comes from PyPI, you may have to do a `pip uninstall pywb && pip install pywb` to get it to update properly.

Small changes:

* You can now download the WARC for a link with type=warc_download
* use a local copy of jquery.form.min.js rather than loading it from their git repo.
* Remove a bunch of redundant inclusions of css/font-awesome.min.css
* reorganize fabfile.py
* Change install instructions to suggest putting PhantomJS in path, so you can just use brew/apt-get/etc. (although you can still download manually and set PHANTOMJS_BINARY if you want).
* In the default Vagrant setup, you can tail -f all logs (Django/celery/phantomjs/etc.) with `fab logs`.