edsu/bagit — pull request #27 by jobyh (2013-05-24 09:14:13+00:00)
Title: Bagit0.97 support

Support for v0.97 of the BagIt draft specification
=====================================
Based on the [diffs](http://tools.ietf.org/rfcdiff?url2=draft-kunze-bagit-06.txt) between draft 5 (0.96) and [draft 6](http://tools.ietf.org/html/draft-kunze-bagit-06) (0.97) of the BagIt spec - all drafts since 06 are doc date bumps - most recent being draft 09 - the following updates have been made:

* Allowing checksums in manifests to be upper / lower case
* Validating optional tagfiles against checksums in optional tagmanifest-\.txt if they are present
* Metadata (bag-info.txt) items may be repeated - also added support for this in make_bag()
* Updating the BagIt spec version to 0.97 where appropriate

The updates shouldn't affect any bags < 0.97 for example the following bag (with an intentionally bad tagmanifest checksum for optional tagfile mytags/mytagfile):

```
testbag
├── bag-info.txt
├── bagit.txt
├── data
│ └── index.html
├── manifest-md5.txt
├── mytags
│ └── mytagfile
└── tagmanifest-md5.txt
```
With bagit.txt BagIt-Version: 0.97
```python
>>> import bagit
>>> bag = bagit.Bag('testbag')
>>> bag.is_valid()
WARNING:root:testbag/mytags/mytagfile: stored hash NOTAHASH doesn't match calculated hash 0026d0808b6a9d637565d0c8d78dbe49
False
```
With bagit.txt BagIt-Version: 0.96 the optional tagfile is ignored
```python
>>> import bagit
>>> bag = bagit.Bag('testbag')
>>> bag.is_valid()
True
```

Checksum Case
-----------------------
At present if a manifest checksum contains uppercase letters it will fail comparison with the internally generated hexdigest (hashlib.hexdigest() outputs with lowercase letters). To prevent this happening the manifest checksum is forced to lowercase at the point of comparison.
jobyh@ ac74ba4d7e7f71f6acb80fe768cfaaebd6e58e02

Optional tagfiles/directories & validation
------------------------------------------------------
Currently extraneous files and directories are ignored which is the desired behaviour under 0.97. However the updated definition of valid requires that any non-standard tagfile entries in optional tagmanifest-\.txt files in the base directory are verified against the corresponding checksum in order for the bag to be considered valid.

This is implemented by adding any files found in the optional tagmanifest-\.txt files to the bag.entries dict so that they can be validated along with the payload entries. As this means that the entries dict no longer necessarily exclusively contains payload entries the utility methods bag.payload_entries() and tagfile_entries() have been added which return a dict of payload entries or (optional) tagfile entries respectively.

As these tagfiles/directories are entirely optional and structure / naming is arbitrary I've not added any make_bag() functionality for optional tagfiles / tagmanifests - seemed overcomplicated and unnecessary at the present time.

Repetition of metadata items
---------------------------------------
At present calling bagit.Bag(\) where \ contains a bag-info.txt with multiple entries for a particular header reassigns the corresponding value to the internal bag.entries dict so only the last entry for a given metadata header ends up being stored in the returned object.

Under the updated code when repeated tag headers are encountered the corresponding value for that key in the bag.info dict is converted to a list and values appended to this in the order they are parsed as the 0.97 spec indicates that we should assume that the ordering is significant.

**Note:** *Although it's clearly a daft idea in the case of Payload-Oxum the spec doesn't prohibit the repetition of any meta items - which is why a test for multiple Payload-Oxum values is added (we use the first encountered value from the stored list - again as we are to assume ordering is significant).*

The 0.96 version of the spec does not discuss repetition of metadata in bag-info.txt either way so the above list behaviour is also left in place for v0.96 bags.