Package: hidebound
IntroductionA local database service for converting directories of arbitrary files into
validated assets and derived metadata for export to databases like AWS S3 and
MongoDB.Seedocumentationfor details.InstallationPythonpip install hideboundDockerInstalldocker-desktopdocker pull thenewflesh/hidebound:latestDocker For DevelopersInstalldocker-desktopEnsure docker-desktop has at least 4 GB of memory allocated to it.git clone git@github.com:theNewFlesh/hidebound.gitcd hideboundchmod +x bin/hideboundbin/hidebound startThe service should take a few minutes to start up.Runbin/hidebound --helpfor more help on the command line tool.OverviewHidebound is an ephemeral database and asset framework used for generating,
validating and exporting assets to various data stores. Hidebound enables
developers to ingest arbitrary sets of files and output them as content and
generated metadata, which has validated according to specifications they define.Assets are placed into an ingress directory, typically reserved for Hidebound
projects, and then processed by Hidebound. Hidebound extracts metadata from the
files and directories that make each asset according to their name, location and
file properties. This data comprises the entirety of Hidebound's database at any
one time.DataflowData begins as files on disk. Hidebound creates a JSON-compatible dict from
their name traits and file traits and then constructs an internal database table
from them, one dict per row. All the rows are then aggregated by asset, and
converted into JSON blobs. Those blobs are then validated according to their
respective specifications. Files from valid assets are then copied or moved into
Hidebound's content directory, according to their same directory structure and
naming. Metadata is written to JSON files inside Hidebound's metadata directory.
Each file's metadata is written as a JSON file in /hidebound/metadata/file, and
each asset's metadata (the aggregate of its file metadata) is written to
/hidebound/metadata/asset. From their exporters, can export the valid
asset data and its accompanying metadata to various locations, like an AWS S3
bucket.WorkflowThe acronynm to remember for workflows isCRUDES: create, read, update,
delete, export and search. Those operations constitue the main functionality
that Hidebound supports.Create AssetFor example, an asset could be an image sequence, such as a directory full of
PNG files, all of which have a frame number, have 3 (RGB) channels, and are 1024
pixels wide by 1024 pixels tall. Let's call the specification for this type of
asset "spec001". We create an image sequence of a cat running, and we move it
into the Hidebound projects directory.UpdateWe call the update function via Hidebound's web app. Hidebound creates
a new database based upon the recursive listing of all the files within said
directory. This database is displayed to us as a table, with one file per row.
If we choose to group by asset in the app, the table will display one asset per
row. Hidebound extracts metadata from each filename (not any directory name) as
well as from the file itself. That metadata is called file_traits. Using only
information derived from filename and file traits, Hidebound determines which
files are grouped together as a single asset and the specification of that
asset. Asset traits are then derived from this set of files (one or more).
Finally, Hidebound validates each asset according to its determined
specification. All of this data is displayed as a table within the web app.
Importantly, all of the errors in filenames, file traits and asset traits are
included.Review GraphIf we click on the graph tab, we are greeted by a hierarchical graph of all our
assets in our project directory. Our asset is red, meaning it's invalid. Valid
asset's are green, and all other files and directories, including parent
directories, are cyan.Diagnose and RepairWe flip back to the data tab. Using table within it, we search (via SQL) for our
asset within Hidebound's freshly created database. We see an error in one of the
filenames, conveniently displayed in red text. The descriptor in one orf our
filenames has capital letters in it. This violates Hidebound's naming
convention, and so we get an error. We go and rename the file appropriately and
call update again.  Our asset is now valid. The filenames are correct and we can
see in the height and width columns, that it's 1024 by 1024 and the channels
column says it has three.CreateNext we click the create button. For each valid asset, Hidebound generates file
and asset metadata as JSON files within the hidebound/metadata directory.
Hidebound also copies or moves, depending on the config write mode, valid files
and directories into the hidebound/content directory. Hidebound/content and
hidebound/metadata are both staging directories used for generating a valid
ephemeral database. We now have a hidebound directory that looks like this
(unmentioned assets are collapsed behind the ellipses):/tmp/hidebound
├──hidebound_config.yaml
│
├──specifications
│└──specifications.py
│
├──data
│...
│└──p-cat001
│└──spec001
│└──p-cat001_s-spec001_d-running-cat_v001
│├──p-cat001_s-spec001_d-running-cat_v001_c0000-0005_f0001.png
│├──p-cat001_s-spec001_d-running-cat_v001_c0000-0005_f0002.png
│└──p-cat001_s-spec001_d-running-cat_v001_c0000-0005_f0003.png
│
├──metadata├──asset│...│└──a9f3727c-cb9b-4eb1-bc84-a6bc3b756cc5.json│└──file...├──279873a2-bfd0-4757-abf2-7dc4f771f992.json├──e50160ae-8678-40b3-b766-ee8311b1f0c9.json└──ea95bd79-cb8f-4262-8489-efe734c5f65c.jsonExportThe hidebound directories contain only valid assets. Thus, we are now free to
export this data to various data stores, such as AWS S3, MongoDB, and Girder.
Exporters are are defined within the exporters subpackage. They expect a
populated hidebound directory and use the files and metadata therein to export
hidebound data. Exporter configurations are stored in the hidebound config,
under the "exporters" key. Currently supported exporters include, disk, s3 and
girder. Below we can see the results of an export to Girder in the Girder web
app.DeleteOnce this export process is complete, we may click the delete button. Hidebound
deletes the hidebound/content and hidebound/metdata directories and all their
contents. If write_mode in the Hidebound configuration is set to "copy", then
this step will merely delete data created by Hidebound. If it is set to "move",
then Hidebound will presumably delete, the only existing copy of out asset data
on the host machine. The delete stage in combination with the removal of assets
from the ingress directory is what makes Hidebound's database ephemeral.Workflow/api/workflowis a API endpoint that initializes a database a with a given
config, and then calls each method from a given list. For instance, if you send
this data to/api/workflow:{config={...}, workflow=['update', 'create', 'export', 'delete']}A database instance will be created with the given config, and then that
instance will call its update, create, export and delete methods, in that order.Naming ConventionHidebound is a highly opinionated framework that relies upon a strict but
composable naming convention in order to extract metadata from filenames. All
files and directories that are part of assets must conform to a naming
convention defined within that asset's specification.In an over-simplified sense; sentences are constructions of words. Syntax
concerns how each word is formed, grammar concerns how to form words into a
sentence, and semantics concerns what each word means. Similarly, filenames can
be thought of as crude sentences. They are made of several words (ie fields).
These words have distinct semantics (as determines by field indicators). Each
word is constructed according to a syntax (ie indicator + token). All words are
joined together by spaces (ie underscores) in a particular order as determined
by grammar (as defined in each specification).SyntaxNames consist of a series of fields, each separated by a single underscore
“_”, also called a field separator.Periods, ".", are the exception to this, as it indicates file extension.Legal characters include and only include:NameCharactersUseUnderscore_only for field separationPeriod.only for file extensionsLowercase lettera to zeverythingNumber0 to 9everythingHyphen-token separatorFields are comprised of two main parts:NameUseField indicatordetermines metadata keyField tokena set of 1+ characters that define the field's dataExample DiagramsIn our example filename:p-cat001_s-spec001_d-running-cat_v001_c0000-0005_f0003.pngthe metadata will be:{"project":"cat001","specification":"spec001","descriptor":"running-cat","version":1,"coordinate":[0,5],"frame":3,"extension":"png",}The spec001 specification is derived from the second field of this filename:fieldfieldindicatortoken|__|__|||p-cat001_s-spec001_d-running-cat_v001_c0000-0005_f0003.png|_______||fieldPartValueFields-spec001Field indicators-Field tokenspec001Derived metadata{specification: spec001}Special Field SyntaxProjects begin with 3 or 4 letters followed by 1 to 4 numbersSpecifications begin with 3 or 4 letters followed by 3 numbersDescriptors begin with a letter or number and may also contain hyphensDescriptors may not begin with the words master, final or lastVersions are triple-padded with zeros and must be greater than 0Coordinates may contain up to 3 quadruple-padded numbers, separated by hyphensCoordinates are always evaluated in XYZ order. For example:c0001-0002-0003produces{x: 1, y: 2, z: 3}.Each element of a coordinate may be equal to or greater than zeroFrames are quadruple-padded and are greater than or equal to 0Extensions may only contain upper and lower case letters a to z and numbers 0
to 9SemanticsHidebound is highly opionated, especially with regards to its semantics. It
contains exactly seven field types, as indicated by their field indicators.
They are:FieldIndicatorprojectp-specifications-descriptord-versionvcoordinatecframefextension.GrammarThe grammar is fairly simple:Names are comprised of an ordered set of fields drawn from the seven aboveAll names must contain the specification fieldAll specification must define a field orderAll fields of a name under that specification must occcur in its defined
field orderIts is highly encouraged that fields be defined in the following order:project specification descriptor version coordinate frame extensionThe grammatical concept of field order here is one of rough encapsulation:Projects contain assetsAssets are grouped by specificationA set of assets of the same content is grouped by a descriptorThat set of assets consists of multiple versions of the same contentA single asset may broken into chunks, identified by 1, 2 or 3 coordinatesEach chunk may consist of a series of files seperated by frame numberEach file has an extensionEncouraged Lexical ConventionsSpecifications end with a triple padded number so that they may be explicitely
versioned. You redefine an asset specification to something slightly
different, by copying its specification class, adding one to its name and
change the class attributes in some way. That way you always maintain
backwards compatibility with legacy assets.Descriptors are not a dumping ground for useless terms like wtf, junk, stuff,
wip and test.Descriptors should not specify information known at the asset specification
level, such as the project name, the generic content of the asset (ie image,
mask, png, etc).Descriptors should not include information that can be known from the
preceding tokens, such as version, frame or extension.A descriptor should be applicable to every version of the asset it designates.Use of hyphens in descriptors is encouraged.When in doubt, hyphenate and put into the descriptor.Project StructureHidebound does not formally define a project structure. It merely stipulates
that assets must exist under some particular root directory. Each asset
specification does define a directory structure for the files that make up that
asset. Assets are divided into 3 types: file, sequence and complex. File defines
an asset that consists of a single file. Sequence is defined to be a single
directory containing one or more files. Complex is for assets that consist of an
arbitrarily complex layout of directories and files.The following project structure is recommended:project|--specification|--descriptor|--asset# either a file or directory of files and directories|-fileFor Example/tmp/projects
└──p-cat001├──s-spec002│├──d-calico-jumping││└──p-cat001_s-spec002_d-calico-jumping_v001││├──p-cat001_s-spec002_d-calico-jumping_v001_f0001.png││├──p-cat001_s-spec002_d-calico-jumping_v001_f0002.png││└──p-cat001_s-spec002_d-calico-jumping_v001_f0003.png│││└──d-tabby-playing│├──p-cat001_s-spec002_d-tabby-playing_v001││├──p-cat001_s-spec002_d-tabby-playing_v001_f0001.png││├──p-cat001_s-spec002_d-tabby-playing_v001_f0002.png││└──p-cat001_s-spec002_d-tabby-playing_v001_f0003.png│││└──p-cat001_s-spec002_d-tabby-playing_v002│├──p-cat001_s-spec002_d-tabby-playing_v002_f0001.png│├──p-cat001_s-spec002_d-tabby-playing_v002_f0002.png│└──p-cat001_s-spec002_d-tabby-playing_v002_f0003.png│└──spec001└──p-cat001_s-spec001_d-running-cat_v001├──p-cat001_s-spec001_d-Running-Cat_v001_c0000-0005_f0002.png├──p-cat001_s-spec001_d-running-cat_v001_c0000-0005_f0001.png└──p-cat001_s-spec001_d-running-cat_v001_c0000-0005_f0003.pngApplicationThe Hidebound web application has five sections: data, graph, config, api and
docs.DataThe data tab is the workhorse of the Hidebound app.Its functions are as follows:Search - Search the updated database's data via SQLDropdown - Groups search results by file or assetInit - Initialized the database with the current configUpdate - Initializes and updates the database with the current configCreate - Copies or moves valid assets to hidebound/content directory and
creates JSON files in hidebound/metadata directoryDelete - Deletes hidebound/content and hidebound/metadata directoriesPrior to calling update, the application will look like this:GraphThe graph tab is used for visualizing the state of all the assets within a root
directory.It's color code is as follows:ColorMeaningCyanNon-asset file or directoryGreenValid assetRedInvalid assetConfigThe config tab is used for uploading and writing Hidebound's configuration file.APIThe API tab is really a link to Hidebound's REST API documentation.DocsThe API tab is really a link to Hidebound's github documentation.ErrorsHidebound is oriented towards developers and technically proficient users. It
displays errors in their entirety within the application.ConfigurationHidebound is configured via a configuration file or environment variables.Hidebound configs consist of four main sections:Baseingress_directory - the directory hidebound parses for assets that comprise its databasestaging_directory - the staging directory valid assets are created inspecification_files - a list of python specification filesinclude_regex - filepaths in the root that match this are included in the databaseexclude_regex - filepaths in the root that match this are excluded from the databasewrite_mode - whether to copy or move files from root to stagingredact_regex - regular expression which matches config keys whose valuse are to be redactedredact_hash - whether to redact config values with "REDACTED" or a hash of the valueworkflow - order list of steps to be followed in workflowDaskDefault configuration of Dask distributed framework.
cluster_type - dask cluster type
num_partitions - number of partions for each dataframe
local_num_workers - number of workers on local cluster
local_threads_per_worker - number of threads per worker on local cluster
local_multiprocessing - use multiprocessing for local cluster
gateway_address - gateway server address
gateway_proxy_address - scheduler proxy server address
gateway_public_address - gateway server address, as accessible from a web browser
gateway_auth_type - authentication type
gateway_api_token - api token
gateway_cluster_options - list of dask gateway cluster options
gateway_shutdown_on_close - whether to shudown cluster upon closeExportersWhich exporters to us in the workflow.
Options include:s3diskgirderWebhooksWebhooks to call after the export phase has completed.Environment VariablesIfHIDEBOUND_CONFIG_FILEPATHis set, Hidebound will ignore all other
environment variables and read the given filepath in as a yaml or json config
file.VariableFormatPortionHIDEBOUND_CONFIG_FILEPATHstrEntire Hidebound config fileHIDEBOUND_INGRESS_DIRECTORYstringress_directory parameter of configHIDEBOUND_STAGING_DIRECTORYstrstaging_directory parameter of configHIDEBOUND_INCLUDE_REGEXstrinclude_regex parameter of configHIDEBOUND_EXCLUDE_REGEXstrexclude_regex parameter of configHIDEBOUND_WRITE_MODEstrwrite_mode parameter of configHIDEBOUND_REDACT_REGEXstrredact_regex parameter of configHIDEBOUND_REDACT_HASHstrredact_hash parameter of configHIDEBOUND_WORKFLOWyamlworkflow paramater of configHIDEBOUND_SPECIFICATION_FILESyamlspecification_files section of configHIDEBOUND_DASK_CLUSTER_TYPEstrdask cluster typeHIDEBOUND_DASK_NUM_PARTITIONSintnumber of partions for each dataframeHIDEBOUND_DASK_LOCAL_NUM_WORKERSintnumber of workers on local clusterHIDEBOUND_DASK_LOCAL_THREADS_PER_WORKERintnumber of threads per worker on local clusterHIDEBOUND_DASK_LOCAL_MULTIPROCESSINGstruse multiprocessing for local clusterHIDEBOUND_DASK_GATEWAY_ADDRESSstrgateway server addressHIDEBOUND_DASK_GATEWAY_PROXY_ADDRESSstrscheduler proxy server addressHIDEBOUND_DASK_GATEWAY_PUBLIC_ADDRESSstrgateway server address, as accessible from a web browserHIDEBOUND_DASK_GATEWAY_AUTH_TYPEstrauthentication typeHIDEBOUND_DASK_GATEWAY_API_TOKENstrapi tokenHIDEBOUND_DASK_GATEWAY_CLUSTER_OPTIONSyamllist of dask gateway cluster optionsHIDEBOUND_DASK_GATEWAY_SHUTDOWN_ON_CLOSEstrwhether to shudown cluster upon closeHIDEBOUND_EXPORTERSyamlexporters section of configHIDEBOUND_WEBHOOKSyamlwebhooks section of configHIDEBOUND_TESTINGstrrun in test modeConfig FileHere is a full example config with comments:ingress_directory:/mnt/storage/projects# where hb looks for assetsstaging_directory:/mnt/storage/hidebound# hb staging directoryinclude_regex:""# include files that matchexclude_regex:"\\.DS_Store"# exclude files that matchwrite_mode:copy# copy files from root to staging# options: copy, moveredact_regex:"(_key|_id|_token|url)$"# regex matched config keys to redactredact_hash:true# hash redacted valuesworkflow:# workflow steps-delete# clear staging directory-update# create database from ingress files-create# stage valid assets-export# export assets in stagingspecification_files:# list of spec files-/mnt/storage/specs/image_specs.py-/mnt/storage/specs/video_specs.pydask:cluster_type:local# Dask cluster type# options: local, gatewaynum_partitions:16# number of partions for each dataframelocal_num_workers:16# number of workers on local clusterlocal_threads_per_worker:1# number of threads per worker on local clusterlocal_multiprocessing:true# use multiprocessing for local clustergateway_address:http://proxy-public/services/dask-gateway# gateway server addressgateway_proxy_address:gateway://traefik-daskhub-dask-gateway.core:80# scheduler proxy server addressgateway_public_address:https://dask-gateway/services/dask-gateway/# gateway server address, as accessible from a web browsergateway_auth_type:jupyterhub# authentication typegateway_api_token:token123# api tokengateway_cluster_options:# list of dask gateway options-field:image# option fieldlabel:image# option labeloption_type:select# options: bool, float, int, mapping, select, stringdefault:"some-image:latest"# option default valueoptions:# list of choices if option_type is select-"some-image:latest"# choice 1-"some-image:0.1.2"# choice 2gateway_shutdown_on_close:true# whether to shudown cluster upon closeexporters:# dict of exporter configs-name:disk# export to disktarget_directory:/mnt/storage/archive# target locationmetadata_types:# options: asset, file, asset-chunk, file-chunk-asset# only asset and file metadata-filedask:# dask settings overridenum_workers:8local_threads_per_worker:2-name:s3# export to s3access_key:ABCDEFGHIJKLMNOPQRST# aws access keysecret_key:abcdefghijklmnopqrstuvwxyz1234567890abcd# aws secret keybucket:prod-data# s3 bucketregion:us-west-2# bucket regionmetadata_types:# options: asset, file, asset-chunk, file-chunk-asset# drop file metadata-asset-chunk-file-chunkdask:# dask settings overridecluster_type:gatewaynum_workers:64-name:girder# export to girderapi_key:eyS0nj9qPC5E7yK5l7nhGVPqDOBKPdA3EC60Rs9h# girder api keyroot_id:5ed735c8d8dd6242642406e5# root resource idroot_type:collection# root resource typehost:http://prod.girder.com# girder server urlport:8180# girder server portmetadata_types:# options: asset, file-asset# only asset metadatadask:# dask settings overridenum_workers:10dask:# dask settings overridenum_workers:10webhooks:# call these after export-url:https://hooks.slack.com/services/ABCDEFGHI/JKLMNO# slack URLmethod:post# post this to slacktimeout:60# timeout after 60 seconds# params: {}                                                         # params to post (NA here)# json: {}                                                           # json to post (NA here)data:# data to postchannel:"#hidebound"# slack datatext:export complete# slack datausername:hidebound# slack dataheaders:# request headersContent-type:application/jsonSpecificationAsset specifications are defined in python using the base classes found in
specification_base.py. The base classes are defined using the schematics
framework. Hidebound generates a single JSON blob of metadata for each file of
an asset, and then combines blob into a single blob with a list values per key.
Thus every class member defined with schematics is encapsulated with ListType.Example assetSuppose we have an image sequence asset that we wish to define a specificqtion
for. Our image sequences consist of a directory containing 1 or 3 channel png
with frame numbers in the filename.projects└──cat001└──raw001└──p-cat001_s-raw001_d-calico-jumping_v001├──p-cat001_s-raw001_d-calico-jumping_v001_f0001.png├──p-cat001_s-raw001_d-calico-jumping_v001_f0002.png└──p-cat001_s-raw001_d-calico-jumping_v001_f0003.pngExample specificationWe would write the following specification for such an asset.fromschematics.typesimportIntType,ListType,StringTypeimporthidebound.core.validatorsasvd# validates traitsimporthidebound.core.traitsastr# gets properties of files and file namesfromhidebound.core.specification_baseimportSequenceSpecificationBaseclassRaw001(SequenceSpecificationBase):asset_name_fields=[# naming convention for asset directory'project','specification','descriptor','version']filename_fields=[# naming convention for asset files'project','specification','descriptor','version','frame','extension']height=ListType(IntType(),required=True)# heights of png imageswidth=ListType(IntType(),required=True)# widths of png imagesframe=ListType(IntType(),required=True,validators=[vd.is_frame]# validates that frame is between 0 and 9999)channels=ListType(IntType(),required=True,validators=[lambdax:vd.is_in(x,[1,3])]# validates that png is 1 or 3 channel)extension=ListType(StringType(),required=True,validators=[vd.is_extension,lambdax:vd.is_eq(x,'png')# validates that image is png])file_traits=dict(width=tr.get_image_width,# retrieves image width from fileheight=tr.get_image_height,# retrieves image height from filechannels=tr.get_num_image_channels,# retrieves image channel number from file)