Package: cdb
Container Database (cdb)This is the Python support tool forcontainerdbto support generation ofdata containers.
Python is more friendly to generating arbitrary data structures, and is popular among the
data science community, so I chose it for metadata generation instead of using GoLang.Have your data and use it too!For documentation and full examples seevsoch.github.io/cdb. These
examples are also available in theexamplesfolder.Getting StartedWhat is a Data Container?A data container is generally an operating-system-less container that is optimized
to provide data, either for query/search, or binding for analysis. The qualities of
the data container should be:It can be mounted to containers with operating systems to run analysisIt can be interacted with on it's own to search metadata about the dataIt should not have an operating system.How do we generate one?The generation is fairly simple! It comes down to a three step multistage build:Step 1We installcdbto generate a GoLang template for anin-memory databasefor our data)Step 2We compile the binary into an entrypointStep 3We add the data and the binary entrypoint to a scratch container (no operating system).And then we interact with it! This tutorial will show you the basic steps to
perform the multistage-build using a simpleDockerfilealong with the data folder. TheDockerfilein the base of the repository also is a good example.UsageDocker UsageThe intended usage is via Docker, so you don't need to worry about installation of
Python, GoLang, and multistage builds to basically:Generate a db.go templateCompile itAdd to scratch with data as data container entrypoint.Thus, to run the dummy example here using theDockerfile:$dockerbuild-tdata-container.We then have a simple way to do the following:metadataIf we just run the container, we get a listing of all metadata alongside the key.$dockerrunentrypoint/data/avocado.txt{"size":9,"sha256":"327bf8231c9572ecdfdc53473319699e7b8e6a98adf0f383ff6be5b46094aba4"}/data/tomato.txt{"size":8,"sha256":"3b7721618a86990a3a90f9fa5744d15812954fba6bb21ebf5b5b66ad78cf5816"}listWe can also just list data files with-ls$dockerrunentrypoint-ls
/data/avocado.txt
/data/tomato.txtorderbyOr we can list ordered by one of the metadata items:$dockerrunentrypoint-metricsize
Orderbysize
/data/tomato.txt:{"size":8,"sha256":"3b7721618a86990a3a90f9fa5744d15812954fba6bb21ebf5b5b66ad78cf5816"}/data/avocado.txt:{"size":9,"sha256":"327bf8231c9572ecdfdc53473319699e7b8e6a98adf0f383ff6be5b46094aba4"}searchOr search for a specific metric based on value.$dockerrunentrypoint-metricsize-search8/data/tomato.txt8$dockerrunentrypoint-metricsha256-search8/data/avocado.txt327bf8231c9572ecdfdc53473319699e7b8e6a98adf0f383ff6be5b46094aba4
/data/tomato.txt3b7721618a86990a3a90f9fa5744d15812954fba6bb21ebf5b5b66ad78cf5816getOr we can get a particular file metadata by it's name:$dockerrunentrypoint-get/data/avocado.txt
/data/avocado.txt{"size":9,"sha256":"327bf8231c9572ecdfdc53473319699e7b8e6a98adf0f383ff6be5b46094aba4"}or a partial match:$dockerrunentrypoint-get/data/
/data/avocado.txt{"size":9,"sha256":"327bf8231c9572ecdfdc53473319699e7b8e6a98adf0f383ff6be5b46094aba4"}/data/tomato.txt{"size":8,"sha256":"3b7721618a86990a3a90f9fa5744d15812954fba6bb21ebf5b5b66ad78cf5816"}startThe start command is intended to keep the container running, if we are using
it with an orchestrator.$dockerrundata-container-startOrchestrationIt's more likely that you'll want to interact with files in the container via
some analysis, or more generally, another container. Let's put together
a quickdocker-compose.ymlto do exactly that.version: "3"
services:
  base:
    restart: always
    image: busybox
    entrypoint: ["tail", "-f", "/dev/null"]
    volumes:
      - data-volume:/data

  data:
    restart: always
    image: data-container
    command: ["-start"]
    volumes:
      - data-volume:/data

volumes:
  data-volume:Notice that the command for the data-container to start is-start, which
is important to keep it running. After building ourdata-container, we can then bring these containers up:$docker-composeup-d
Startingdocker-simple_base_1...doneRecreatingdocker-simple_data_1...done$docker-composepsNameCommandStatePorts
---------------------------------------------------------
docker-simple_base_1tail-f/dev/nullUpdocker-simple_data_1/entrypoint-startUpWe can then shell inside and see our data!$dockerexec-itdocker-simple_base_1sh
/# ls /data/avocado.txttomato.txtThe metadata is still available for query by interacting with the data-container
entrypoint:$dockerexecdocker-simple_data_1/entrypoint-ls
/data/avocado.txt
/data/tomato.txtDepending on your use case, you could easily make this available inside the
other container. This is very simple usage, but the idea is powerful! We can interact with the dataset
and search it without needing an operating system. It follows that we can develop
customized data-containers based on the format / organization of the data inside
(e.g., a data-container that knows how to expose inputs, outputs, etc.)Python UsageThe above doesn't require you to install the Container Database (cdb) metadata
generator, however if you want to (to develop or otherwise interact) you
can do the following. First, install cdb from pypi or a local repository:$pipinstallcdborgitclonegit@github.com:vsoch/cdbcdcdb
pipinstall-e.Command LineThe next step is to generate the goLang file to compile.
You'll next want to change directory to somewhere you have a dataset folder.
For example, intestswe have a dummy "data" folder.cdtests/We might then runcdb generateto create a binary for our container, targeting
thetests/datafolder:$cdbgeneratedata--outdb.goThe db.go file is then in the present working directory. You can either
build it during a multistage build as is done in theDockerfile,
or do it locally with your own GoLang install and then add to the container.
For example, to compile:gogetgithub.com/vsoch/containerdb&&\GOOS=linuxGOARCH=amd64gobuild-ldflags="-w -s"-o/db-i/db.goAnd then a very basic Dockerfile would need to add the data at the path specified,
and the compiled entrypoint.FROMscratch
WORKDIR/data
COPYdata/.
COPYdb/db
CMD["/db"]A more useful entrypoint will be developed soon! This is just a very
basic start to the library.PythonYou can run the same generation functions interactively with Python.fromcdb.mainimportContainerDatabasedb=ContainerDatabase(path="data")# <cdb.main.ContainerDatabase at 0x7fcaa9cb8950>View that there is a files generator atdb.filesdb.files<generatorobjectrecursive_findat0x7fcaaa4ae950>And then generate! If you don't provide an output file, a string will be returned.
Otherwise, the output file name is returned.output=db.generate(output="db.go",force=True)Currently, functions for parsing metadata are named incdb/functions.py,
however you can also define a custom import path. This has not yet been tested
and will be soon. We will also be added more real world examples soon.LicenseFree software: MPL 2.0 License