mytardis/mytardis — pull request #221 by crawley (2013-03-11 23:51:48+00:00)
Title: Datafile migration part 2

This is part 2 of the datafile migration changes. The big picture items:

Core model changes:
------------------------------

- Each Datafile now has 1 or more Replicas which holds the url, protocol, validation and stayRemote fields previously on Datafile.
- Every Replica is related to a Location which gives a location base_url, a type (online, offline, external), a transfer provider (currently local filesystem, plain http or webdav) and a set of provider-specific parameters.
- The South migrations for these changes involve a schema and data migrations that are NOT reversible. BEWARE!! BACK UP YOUR DATABASES.

Core functionality changes:
---------------------------------------

- Everything (as far as I can tell) in the codebase has been changed to match the model changes.
- There is now code in the model to (optionally) enforce the constraints that every Datafile has a recorded checksum and length. This is ON by default ...
- Fetching, storing and moving of data is now done using TransferProviders. The idea is roughly the same as the Django Storage abstraction ... but it works on Replicas (not string paths) and provides an extra verb for fetching file metadata (checksum / length) for file transfer validation.
- Support for the old deprecated "non-mets" XML ingestion format is gone!

Migration app changes:
---------------------------------

- Model changes required complete overhaul
- The old "provider" code has been replaced with stuff in core (see above)
- The "migrate" subcommand is symmetric ... you can migrate from any online Location to any other. (The old "restore" subcommand is unnecessary and removed.)

Other:
---------
- Documentation updates done.
- Unit tests run clean