Package: dabapush
dabapushDatabase pusher for social media data (Twitter for the beginning) – pre-alpha versionUsing dabapushdabapushis a tool to read longer running data collections and write them to another file format or persist them into a database. It is designed to run periodically, e.g. controlled by chron, thus, for convenience ot use project-based configurations which contain all required information on what to read where and what to do with it.
Aprojectmay have one or morejobs, each job consists of a reader and a writer configuration, e.g. read JSON-files from the Twitter API that we stored in folder/home/user/fancy-project/twitter/and write the flattened and compiled data set in to/some/where/elseas CSV files.First stepsIn order to run a firstdabapush-job we'll need to create a project configuration. This is done by calling:dabapushcreateBy default this walks you through the configuration process in a step-by-step manner. Alternatively, you could call:dabapushcreate--non-interactiveThis will create an empty configuration, you'll have to fill out the required information by e.g. calling:dabapushreaderaddNDJSONdefault
dabapushwriteraddCSVdefaultWhereasreader add/writer addis the verb,NDJSONorCSVis the plugin to add anddefaultis the pipeline name.Of course you can edit the configration after creation in your favorite editor, butBEWARE NOT TO TEMPER WITH THE YAMl-TAGS!!!.To run the newly configured job, please call:dabapushrundefaultCommand ReferenceInvocation Patterndabapush<command><subcommand?><options>Commandscreate-- creates a dabapush project (invokes interactive prompt)Options:--non-interactive, create an empty configuration and exit--interactive,this is the default behavior: prompts for user input onproject name,project authors name,project author email address(es) for notificationsmanually configure targets or run discover?run all-- collect all known items and execute targets/destinationsrun <target>-- run a single writer and/or named targetOptions:--force-rerun, -r: forces all data  to be read, ignores already logged datareader-- interact with readersreader configure <name>-- configure the reader for one or more subproject(s); Reader configuration is inherited from global to local level; throws if configuration is incomplete and defaults are missingreader list: returns a table of all configured readers, with<path> <target> <class> <id>reader list_all: returns a table of all registered reader pluginsreader add <type> <name>: add a reader to the project configurationOptions:--input-directory <path>: directory to be read--pattern <pattern>: pattern for matching file names against.remove <name>: remove a reader from the project configuration.register <path>: not there yetdiscover-- discover (possible) targets in project directory and configure them automagically -- yeah, you dream of that, don't you?writer-- interact with writerswriter add <type> <name>:writer remove <name>: removes the writer for the given namewriter list-- returns table of all writers, with<path> <subproject-name> <class> <id>writer list_all: returns a table of all registered writer pluginswriter configure <name>orwriter configure allOptions:--output-dir, -o <path>: default for all targets:<project-dir>/output/<target-name>--output-pattern, -p <pattern>: pattern used for file name creation e.g. 'YYYY-MM-dd', file extension is added by the writer and cannot be overwritten--roll-over, -r ``<file-size>:--roll-over, -r<lines>:--roll-over -r <None>: should be the output chunked? Give either a file-size or a number of lines for roll-over or None to disable chunkingExtending dabapush and developers guideDabapush's reader and writer plug-ins are registered via entry point:dabapush_readersfor readers anddabapush_writersfor writers. Both expectConfiguration-subclass.Developer InstallationInstallpoetryClone repositoryIn the cloned repository's root directory runpoetry installRunpoetry shellto start development virtualenvRundabapush createto create your first project.Runpytestto run all tests