mytardis/mytardis — pull request #277 by crawley (2013-07-22 01:53:07+00:00)
Title: Archiving

This commit adds "archiving" support to the "migration" app.

An archive consists of a tar/gz file containing all of the data files in an experiment together with a METs format manifest. Archives can be saved to a local directory, or to a Location. In the latter case, we create a record in the (new, app-specific) Archive table to help the administrator keep track of the available archives.

There are two new django commands to support this. The "archive" command creates, transfers and records the archives. It has a few bells and whistles like "--incremental" for only archiving experiments that have changed (as far as we can tell), checksums, transfer verification, "--sendOffline" and hooks for catching archives that are too large or too small. (The last is UQ specific. We archive to a HFS that refuses to write files less than 400k to tape.)

The second command is "archivelist", which is designed to let the admin find out what archives are available.

There are some associated changes to the MyTardis core:
* The sizes of some fields in the Location tables have been increased. This entails a south schema migration.
* The "transfer" API has been expanded and reworked. This is used by "migration" app (and some core code) for moving files between locations.
* There is a new "transfer provider" class that supports file transfer using SCP (with SSH commands to do various related things).