chfoo/wpull — pull request #135 by yipdw (2014-05-30 06:19:48+00:00)
Title: Introduce --move-warc-to for upload pipelines

This PR introduces `--move-warc-to DIR`, which is intended to make wpull fit better in upload pipelines like ArchiveTeam/archiveteam-megawarc-factory. Check out http://badcheese.com/~steve/atlogs/?chan=archiveteam&day=2014-05-30 around the 04:30:00 mark for more information.

The problem: it is currently not easy to determine when wpull has completed a WARC file in a series of WARCs, and we cannot safely upload a sequential WARC while it is being written. The megawarc factory code addresses this problem by requiring atomic mv and watching a directory for new files.

`--move-warc-to DIR` allows completed WARCs to be moved off to an upload staging directory; from that point, scripts like the megawarc uploader can be used as-is.

This PR is not yet production ready; consider this more of a reality check request, i.e. "am I even doing this right?" Here are the problems I know of:

* No tests. I'm still coming to grips with the test suite.
* I'm not sure how this interacts with CDXes or the meta-WARC. (As far as I can tell, it ignores them.) The right thing to do here may be to move the CDX and meta-WARC on job completion to the given directory.
* This option could in theory be applied to any WARC recording situation. This patch does not implement it that way, however.