Package: metastore-lib
metastore-lib: metadata storage library for datapackagesA Python library for abstracting metadata storage fordatapackage.jsonpackages.Full DocumentationWhile this README provides some basic information on how to get started, the
most up-to-date and comprehesive documentation formestastore-libisavailable at metastore-lib.readthedocs.io.InstallationThe easiest way to install the latest stable version of metastore-lib into
your Python environment is viapip:pip install metastore-libQuick StartInstantiating a backendTo use the library after you have installed it, first instantiate a storage
instance:config={"token":"...","more_options":"..."}# Using the provided factory methodmetastore=create_metastore('github',**config)# Or by directly instantiating one of the MetaStoreBackend classes:metastore=GitHubStorage(**config)Storing a dataset (creating a new package)Then use the storage instance to store a dataset:importjsonwithopen("datapackage.json")asf:metadata=json.loads(f)package_info=metastore.create(package_id,metadata)This will store the package metadata using the specific storage backend. For
example, in the case of the GitHub backend, a new repository will be created
with a correspondingdatapackage.jsonfile and LFS pointer files for
resources.The returnedpackage_infowill be an object with some information about
the stored package revision:classPackageRevisionInfo:package_id:str="..."revision:str="..."package:Dict={"name":"mypackage","version":"1.0.0","resources":[# ...]}Updating a datasetTo update the same package:base_rev=package_info.revisionmetadata['version']='1.0.1'package_info=metastore.update(package_id,metadata,base_revision=base_rev)This will update the package, creating a new revision of the metadata. Note thatbase_revisionis not required but is recommended, to ensure changes are not
conflicting; Specifyingbase_revisionwill ensure you are changing based on
the latest revision of the package, and if not aConflictExceptionwill be
raised.Listing Dataset RevisionsNow you can get a list of all revisions of the package (there should be exactly two):revisions=metastore.revision_list(package_id)# Returns: [ <RevisionInfo rev2>, <RevisionInfo rev1> ]Each returned object in the list represents a single revision:classPackageRevisionInfo:package_id:str="..."revision:str="..."created:datetime=...# the revision creation timestampFetching a Dataset RevisionNow that we have two different revisions of the dataset, we can fetch a
specific revision of the metadata:package_info=metastore.fetch(package_id,revision=revisions[0].revision)print(f"{package_info.package['name']}{package_info.package['version']}")# will output: mypackage 1.0.0package_info=metastore.fetch(package_id,revision=revisions[1].revision)print(f"{package_info.package['name']}{package_info.package['version']}")# will output: mypackage 1.0.1This returns aRevisionInfoobject for the requested package / revision.Note that therevisionparameter is optional, and if omitted the latest
revision will be fetched.Creating a TagOnce a revision has been created, you can tag the revision to give it a
meaningful name:tag_info=metastore.tag_create(package_id,revision=revisions[1].revision,name='ver-1.0.1')This will return a newTagInfoobject, with thenameattribute set to'ver-1.0.1'.Listing TagsTo get a list of all tags for a package:tags=metastore.tag_list(package_id)This will return a list ofTagInfoobjects, each pointing to a specific
tagged revision.A Note on Package IdentifiersPackage Identifiers (e.g. thepackage_idin the example above) are strings
and are, as far asmetastoreis concerned, opaque. However, they may still
be meaningful as far as either the backend or the client is concerned.For example, with a GitHub based backend you will use IDs that correlate with<org name>/<repo name>structure.Other backends may expect you to use UUID type identifiers.It is up to the code using themetastorelibrary to be able to compose the
right identifiers.Using the Filesystem Backend for TestingFor testing and quick prototyping purposes, this library offers a specialfilesystembackend, which can be used to save versioned datapackage
information on the file system, in memory or on virtual file system.This backend is based on thePyFilesystemlibrary, and can use any of it's supported file systems as storage.In testing, it is recommended to use a memory based storage:frommetastore.backend.filesystemimportFilesystemStoragedeftest_my_code():"""Test for code that relies on a metastore-lib backend"""backend=FilesystemStorage('mem://')r1=backend.create('some-package',datapackage,'Initial revision')# ... continue with testing ...TheFilesystemStorageconstructor takes a single argument, which is aPyFilesystemroot filesystem URL.Beyond this, all API is exactly the same as with other backends.LicenseCopyright (C) 2020, Viderum, Inc.metastore-lib is free / open source software and is distributed under the terms of
the MIT license. SeeLICENSEfor details.