Package: articat
articatMinimal metadata catalog to store and retrieve metadata about data artifacts.Getting startedAt a high level,articatis simply a key-value store. Value being the Artifact metadata.
Key a.k.a. "Artifact Spec" being:globally uniqueidoptional timestamp:partitionoptional arbitrary string:versionTo publish a file system Artifact (FSArtifact):fromarticatimportFSArtifactfrompathlibimportPathfromdatetimeimportdate# Apart from being a metadata containers, Artifact classes have optional# convenience methods to help in data publishing flow:withFSArtifact.partitioned("foo",partition=date(1643,1,4))asfsa:# To create a new Artifact, always use `with` statement, and# either `partitioned` or `versioned` methods. Use:# * `partitioned(...)`, for Artifacts with explicit `datetime` partition# * `versioned(...)`, for Artifacts with explicit `str` version# Next we produce some local data, this could be a Spark job,# ML model etc.data_path=Path("/tmp/data")data_path.write_text("42")# Now let's stage that data, temporary and final data directories/buckets# are configurable (see below)fsa.stage(data_path)# Additionally let's provide some description, here we could also# save some extra arbitrary metadata like model metrics, hyperparameters etc.fsa.metadata.description="Answer to the Ultimate Question of Life, the Universe, and Everything"To retrieve the metadata about the Artifact above:fromarticat.fs_artifactimportFSArtifactfromdatetimeimportdatefrompathlibimportPath# To retrieve the metadata, use Artifact object, and `fetch` method:fsa=FSArtifact.partitioned("foo",partition=date(1643,1,4)).fetch()fsa.id# "foo"fsa.created# <CREATION-TIMESTAMP>fsa.partition# <CREATION-TIMESTAMP>fsa.metadata.description# "Answer to the Ultimate Question of Life, the Universe, and Everything"fsa.main_dir# Data directory, this is where the data was stored after stagingPath(fsa.joinpath("data")).read_text()# 42Featuresstore and retrieve metadata about your data artifactsno long running services (low maintenance)data publishing utils builtinIO/data format agnosticimmutable metadatadevelopment modeArtifact flavoursCurrently available Artifact flavours:FSArtifact: metadata/utils for files or objects (supports: local FS, GCS, S3 and more)BQArtifact: metadata/utils for BigQuery tablesNotebookArtifact: metadata/utils for Jupyter NotebooksDevelopment modeTo ease development of Artifacts,articatsupports development/dev mode.
Development Artifact can be indicated bydevparameter (preferred), or_devprefix in the Artifactid. Dev mode supports:overwriting Artifact metadataconfigure separate locations (e.g.dev_prefixforFSArtifact), with
potentially different retention periods etcBackendlocal: mostly for testing/demo, metadata is stored locally (configurable, default:~/.config/articat/local)gcp_datastore: metadata is stored in the Google Cloud DatastoreConfigurationarticatconfiguration can be provided in the API, or configuration files. By default configuration
is loaded from~/.config/articat/articat.cfgandarticat.cfgin current working directory. You
can also point at the configuration file via environment variableARTICAT_CONFIG.You uselocalmode without configuration file. Available options:[main]# local or gcp_datastore, default: local# mode =# local DB directory, default: ~/.config/articat/local# local_db_dir =[fs]# temporary directory/prefix# tmp_prefix =# development data directory/prefix# dev_prefix =# production data directory/prefix# prod_prefix =[gcp]# GCP project# project =[bq]# development data BigQuery dataset# dev_dataset =# production data BigQuery dataset# prod_dataset =Our/example setupBelow you can see a diagram of our setup, Articat is just one piece of our system, and solves a specific problem. This should give you an idea where it might fit into your environment: