harvard-lil/perma — pull request #487 by jcushman (2014-04-02 04:12:51+00:00)
Title: Streamline proxy archive capture

This streamlines archive capture by using selenium to let us examine the live page in PhantomJS from Python. So entirely in Python, we can open the warc proxy; load robots.txt in the background; load the page; take an immediate screenshot; fetch the title from the DOM; fetch meta tags from the DOM; take a second screenshot 5 seconds after the first; and close the proxy. This should be more reliable and easier to extend.