Package: tame
tameAn unobtrusive YAML based metadata system for arbitrary data management.RationaleIn many fields, large amounts of interlinked data are generated. Contextual information, such as the experimental layout of the plate used for microscopy or the configuration of a system used to generate a large profile file, is sometimes stored in the relevant file formats. However, the more common case is that this contextual information must be encoded elsewhere.tameaims to be a minimal metadata management system. Metadata is written in the form of YAML files, without defined schemas. These metadata files should always remain human-understandable, even if generated by external tools. In the case of scientific data, these metadata files must often be written by the scientist, which means that complicated non-plaintext formats or even heavily-delimited plain-text formats (ala JSON) are untenable. At the same time, having this metadata be computer-readable is a boon; relations between files can be programmatically queried, used, and modified.Basic useWriting metadata filesFor each piece data you want to attach metadata to, write a YAML document.tameonly looks for one required tag for each metadata file, atypekey. Normally, thefileskey is also present, unless there is a piece of data that is metadata-only. Optionally, the keysnameanduidcan be included and have a special meaning fortame. Other keys may be present and remain tracked bytame.type: A use-specific term describing what information is being tracked. User-written custom code, such as validators or locators, primarily operates on different types.files(required, unless you have a metadata-only document): Which files the metadata is attached to. Paths to files can either beabsoluteorrelative to the location of the YAML document.name(optional): A string describing what the metadata describes. In the absence of auidfield, the name and type of metadata is used for linking pieces of metadata together.tameallows multiple pieces of metadata to have the same type and name.uid(optional): A string referring to a piece of metadata.tameenforces that two pieces of metadatacannothave the sameuidandtype. When theuidis present, it can be used for linking metadata together.Linking metadata togetherMost metadata is not entirely self-contained, and needs contextual information.tamehandles this withparent links. A piece of metadata can be linked to any number of other metadata entries. These links can be specified in one of three ways:By filename. A single key is expected in this case, with valuefile. This directly links a piece of metadata to the named YAML document.By type/name. Two key/value pairs are expected. This attempts to link this piece of metadata to another piece of metadata, looked up by type/name. This can fail if there are multiple pieces of metadata with the same type and name!By type/uid. Becausemetaenforces that type/uid's are unique, there cannot be multiple pieces of metadata with the same type and UID.TODO: locators. How to link to "submetadata" files.See the example below for more details.Links are specified with a YAML list from the keyparent.ExampleImagine a directory layout as follows:â”œâ”€â”€plasmids
â”‚   â”œâ”€â”€â”€GFP.gb
â”‚   â””â”€â”€â”€GFP.yaml
â”œâ”€â”€microscopy
â”‚   â”œâ”€â”€â”€20200101.yaml
â”‚   â”œâ”€â”€â”€green_channel.tif
â”‚   â”œâ”€â”€â”€red_channel.tif
â”‚   â””â”€â”€â”€overlay.tifwith a plasmid YAML file:plasmids/GFP.yaml---
type: plasmid   # Required meta tag
name: GFP       # Optional tag for linking purposes
uid: p001       # Optional tag for linking purposes
files:          # Required meta tag
  - GFP.gb      # This file is specified using a relative path
resistance: Amp # Use-specific tag (in this case a biology metadata field)If the microscopy experiment needs to be linked to the plasmid,
we could specify the linkage in one of these three ways:microscopy/20200101.yaml---
type: image
files:
  - green_channel.tif
  - red_channel.tif
  - overlay.tif
parent:
  - file: ../plasmids/GFP.yaml # Directly specify the parent relationship
  - type: plasmid              # Specify the parent relationship with
    name: GFP                  # a type/name pair, using explicit YAML syntax
  - {type: plasmid, uid: p001} # Specify the parent relationship with inline YAML syntaxSoftware supporttamesupports four basic operations that act on these linked metadata trees,validate,describe,collect, andfreeze.tame validateWhen given a single file,tame validateverifies that the given YAML document is in fact valid YAML. If it is,tamethen verifies that the entire tree of parent relationships is valid.tameraises an error if there is invalid YAML syntax somewhere, if a parent link cannot be found, or if there is a type/name collision given a parent link.When given a directory,tameperforms this validation operation on all reachable YAML files beneath the directory.TODO: How do we ensure that a project that has non-meta YAML files is covered/doesn't error out when we do this? Should we switch the filetype to ".meta" instead of just ".yaml"? That would break the idea that you could use this metadata system even without the programtametame describeGiven a YAML data file, this attempts to "describe" the metadata item, often by giving information derived from itself and its parents. This is often extended by user code to give meaningful information to all of the other key:value pairs thattamedoes not normally care about.tame collectGiven a metadata file, this tellstameto collect tne entire tree of parent links from the given metadata file into a self-contained directory. This collection then has the complete contextual information to be shared elsewhere. In the micrscopy example above, performing atame collecton the GFP.yaml file would only include itself and the GFP plasmid, because this plasmid does not depend on anything else. Performing atame collecton the microscopy YAML file would include both the micrscopy imagesandthe GFP plasmid file, because the parental link means that the image should not be separated from its context of the plasmid from which the cells came from.tame freezeTODO: finish thinking about what this means. The goal is to have a command that would:Expand wildcards into the specifically included files (e.g. if someone wrote *.tif, it actually "freezes" the file include to whatever TIF files are currently present.Hash the included files, saving the hash value, to make sure we have confirmation that files have not been modified.This moves uncomfortably towards a version control system. If you freeze, you should probably save the frozen original files somewhere, as there's no other way to make sure they don't get modified.Finding metadata filesTODO: decide how to do this. Could use a root system, e.g. a tame.yaml file at the top-level, where subdirectories are searched from that point downward.Extendingtameto your metadata schemeFor most usecases, no Python has to be written to extendtame. Just start writing YAML files!tameis extended with Python in two important ways, vialocatorsanddescriptors. At the top-level of the tracked metadata, the user can write Python code that implements this locator and descriptor for eachtypeof metadata.LocatorsLocators allow an additional level of granualartiy in parent links. As an example on where this might be useful, consider a multiple-sample experiment. One metadata YAML file might describe the relevant information of the entire experiment, but there may be additional pieces of data such as micrscopy images that only pertain to single samples. Locators are then the logic that links/matches metadata files together at this sub-file level.TODO: extend with examples/more reasoningDescriptorsDescriptors are given a Python view of a metadata file of a given type, and are responbible for formatting the output of atame describecall. The default descriptor simply prints out the key/value pairs that are not special/reserved fortame.TODO: extend with examples/more reasoningDevelopment setupThis project uses a development strategy inspired bygit-flow. For release sanity and simplicity of non-developers cloning the repository, themasterbranch isalwaysa stable release. Main development occurs on thedevelopbranch, so switch to that branch if you want to make changes!Feel free to ignore the git-flow-ism's of unnecessary merge commits.LicenseThis software is released under theMIT license. This code should be as free as your metadata.