Package: dbispipeline
DBIS PipelineThis pipline can be used to run analyses in a structured way, and stores
configurations and results in a database.Usagethe user writes a minimal plan file which contains only the following
information:"how do I get the data?", by providing a dataloader"what to do with the data?", by providing a scikit pipeline"how to process the result?", by providing result handlers."where to additionally store results?" by providing storage handlers.Please have a look at the examples for more information.CLIWe provide adbispipeline-linktool that can be used to link datasets to
the data directory. This ensures that datasets are linked in a consistent way
even on different machines. The general process is as follows:Either in thedbispipeline.inior as an argument in the cli call, one can
define where in general datasets are stored on the local machine. For example,
many datasets are available on/storage/nas3/datasets/text. In this case,
this would be the value in the configuration:#dbispipeline.ini:
[project]
dataset_dir = /storage/nas3/datasetsIn a filedata/links.yaml, one can define specific datasets that are used
by the software. Thereby, the first path segment will be cut off (not sure why).
For example, the following yaml file:---datasets:-music/acousticbrainz-music/billboard-music/millionsongdatasetwould assume that a physical directory exists at/storage/nas3/datasets/music/billboardand after calling the scriptdbispipeline-linkwithout parameters using the above configuration, the
following symlinks will be created:data/acousticbrainz -> /storage/nas3/datasets/music/acousticbrainz
data/billboard -> /storage/nas3/datasets/music/billboard
data/millionsongdataset -> /storage/nas3/datasets/music/millionsongdatasetThe value ofdataset_dirfrom the config can be overwritten in the cli
script by using the-poption.Requirementspython >= 3.6a PostgreSQL databasean email server if you want to use notification emailsInstallationInstall dbispipeline in your python. We recommend using pipenv to keep your
dependencies clean:pipenv install dbispipelineThis call will install a virtual environment as well as all dependencies.Write your plan(s). See the example plan files for guidance.callpipenv run dp <yourplanfile.py>Enjoy!ConfigurationThe framework look in multiple directories for its configuration files./usr/local/etc/dbispipeline.iniused for system wide default.$HOME/.config/dbispipeline.iniused for user specific configurations../dbispipeline.inifor project specific configurations.And example configuration file looks like this:[database]# url to your postgres databasehost=your.personal.database# your database user nameuser=user# port of your postgres database, default = 5432# port = 5432# password of your database userpassword=<secure-password># database to usedatabase=pipelineresults# table to be usedresult_table=my_super_awesome_results[project]# this will be stored in the databasename=dbispipeline-test# this is used to store backups of the execution# it is possible to override this by setting the DBISPIPELINE_BACKUP_DIR# environment variable# the default is the temp dir of the os if this option is not set.backup_dir=tmp# this is used to linke the used datasets spcified in data/links.yaml# it is possible to override this by setting the DBISPIPELINE_DATASET_DIR# environment variabledataset_dir=/storage/nas/datasets[mail]# email address to use as sendersender=botname@yourserver.com# recipient. This should probably be set on a home-directory-basis.recipient=you@yourserver.com# smtp server address to usesmtp_server=smtp.yourserver.com# use smtp authentication, default = no# authenticate = no# username for smtp authentication, required if authenticate = yes# username = foo# password for smtp authentication, required if authenticate = yes# password = bar# port to use for smtp server connection, default = 465# port = 465