Package: mmmeta
mmmetammmetais a command-line toolkit and python library to incrementally
synchronize file metadata between anarchivethat stores all files
and their metadata,publishersthat add new files to the archive andconsumersthat process these files (or a subset of them).It’s better explained by a concrete example:Publisherincrementally scrapes documents and stores them with
metadata in thearchive.Consumerwants to import some files by a given filter criterion and
keep track of the ones that are already imported.As such file collections grow, we only want to transfer as less data as
possible betweenarchive,publisherandconsumer.synopsisTo clarify the terms used in this manual:files: actual files (like pdfs…)metadata files: json files that contain metadata for actual filesarchive: the “source of truth” where files, metadata files and
metadata db are stored.publisher: an application that adds new files to thearchiveconsumer: an application that processes the files fromarchive(with read-only access)state db: sqlite database, stored only onconsumers, tracking
local state for filesmetadir: a directory named_mmmetathat issyncedbetweenarchive,publishersandconsumersstore: A simple implementation of a key-value store for
additional informationUsageArchiveThe archive can be any file-like (remote) location for the actual files,
theirmetadata filesand themetadir.Publisherswould need
write access to it,Consumersonly need read-only.mmmetausually doesn’t operate on the archive itself (as it would be
most likely just a data bucket), instead, maintaining the archive is
done bypublishersPublisherAn application that writes to the archive. This can be for example a
scraper that incrementally adds new files.The usual workflow would look like this:synchronizemetadirfrom archiveRun application (e.g. scraper) optionally based on synced metadataUpdatemetadir(see below)synchronizemetadirback to archiveupdatingmetadirmmmeta generateThis will loop through all json files in the current directory and
create or add csv data in./_mmmeta/db/For other path locations, seeinitializationmanaging files presencePer default,mmmeta generateonly adds new files based on themetadata filesavailable (it doesn’t even check the presence of the
actual files). To “clean up” (e.g. delete non-existing files), the cli
interface provides the following options:--replace       Completly replace the meta database
--ensure        Ensure metadata files are present, soft-delete non-existing
--ensure-files  Ensure actual files are present (for local store only),
                soft-delete non-existing
--no-meta       Read in actual files instead of json metadata filesConsumerAn application that processes the files, e.g. import them into a
database.mmmetais used to mergeremotemetadata into the localstate db(sqlite) and provides some functionallity to query and manage this data
in applications.The usual workflow would look like this:synchronizemetadirfrom archiveUpdate localstate db(see below)Run application that alters local state (see example below)update local state dbvia cli:mmmeta updateor via python:frommmmetaimportmmmetam=mmmeta()m.update()For other path locations, seeinitializationconsumer applicationThefilesobject on a metadir is a wrapper to adataset
tablewith
all its functionallity, with the addition that it yieldsmmmeta.file.Fileobjects that have a bit extra functionality like
directly saving and access to “proxy values” (see config below)frommmmetaimportmmmetam=mmmeta()forfileinm.files(document_type="contract",imported=False):download_url=file.remote.url# see config belowprocess_download(download_url)file["downloaded"]=Truefile.save()Seeconfigon how to generate remote urls or urisInitializationWhenmmmetaisinitializedwith a path
argument namedfoo, the directoryfoo/_mmmetawill be themetadirThe pathfoocan be set via env var:MMMETA=./foo/ mmmeta updateor in scripts:frommmmetaimportmmmetam=mmmeta("./foo/")Onpublishersthere is an additional env varMMMETA_FILES_ROOTif the location for theactual filesis
different.SynchronizationThis package is totally agnostic about the remote storage backend (could
be a local filesystem location or cloud storage) and doesn’t handle any
of the local <-> remote synchronization.Therefore the synchronization of themetadir./foo/_mmmetais up
to you with the tool of your choice.Configmmmetacan optionally have a config stored in./foo/_mmmeta/config.ymlExample (all settings are optional):metadata:file_name:_file_name# key in json metadat for file nameinclude:# only include these keys from json metadata in meta db-reference-modified_at-title-originators-publisher:name# nested keys are flattened with ":" between themunique:content_hash# unqiue identifier for filesremote:# simple string replacement to generate `File.remote.<attr>` attributes, like:url:https://my_bucket.s3.eu-central-1.amazonaws.com/foo/bar/{_file_name}uri:s3://my_bucket/foo/bar/{_file_name}remoteThe configuration sectionremotefrom above ensures that the file
objects have attributes to access the actual files from the remote:frommmmetaimportmmmetam=mmmeta()forfileinm.files:print(file.remote.uri)Storemmmetaships with a simple key-value-store that can be used by both
theremoteandclientto store some additional data. The store lives
in themetadir./foo/_mmmeta/_storeYou can store any values in it:frommmmetaimportmmmetam=mmmeta("./path/to/metadir/")m.store["new_files"]=17any machine thatsynchronizesthe metadir can
read these values:frommmmetaimportmmmetam=mmmeta("./path/to/metadir/")new_files=m.store["new_files"]# 17For storing timestamps, there is a shorthand via thetouchfunction:m.touch("my_ts_key")This will save the value of the currentdatetime.now()to the keymy_ts_key. The values are typed (int,floatortimestamp), so you can easily do something like this:frommmmetaimportmmmetam=mmmeta("./path/to/metadir/")ifm.store["remote_last_updated"]>m.store["local_last_updated"]:# run scraperInstallationRequires python3. Virtualenv use recommended.Additional dependencies will be installed automatically:pip install mmmetaAfter this, you should be able to execute in your terminal:mmmeta --helpYou should as well be able to import it in your python scripts:frommmmetaimportmmmetacliUsage:mmmeta[OPTIONS]COMMAND[ARGS]...Options:--metadirTEXTBasepathforreadingmetainfoandstoringstate[default:<current/working/dir>]--files-rootTEXTBasepathforactualfilestogeneratemetadirfrom[default:<current/working/dir>]--helpShowthismessageandexit.Commands:generateinspectupdatedevelopementInstall testing requirements:make installTest:make test