Package: dataplaybook
Data Playbook:book: Playbooks for data. Open, process and save table based data.Automate repetitive tasks on table based data. Include various input and output tasks.Install:pip install dataplaybookUse the@taskand@playbookdecoratorsfromdataplaybookimporttask,playbookfromdataplaybook.tasks.io_xlsx@taskdefprintTasksTasks are implemented as simple Python functions and the modules can be found in the dataplaybook/tasks folder.ModuleFunctionsGeneric function to work on tablesdataplaybook.tasksbuild_lookup, build_lookup_var, combine, drop, extend, filter, print, replace, unique, vlookupFuzzy string matchingdataplaybook.taksk.fuzzyRequirespip install fuzzywuzzyRead/write excel files ()dataplaybook.tasks.io_xlsxread_excel, write_excelMisc IO tasksdataplaybook.tasks.io_miscread_csv, read_tab_delim, read_text_regex, wget, write_csvMongoDB functionsdataplaybook.tasks.io_mongoread_mongo, write_mongo, columns_to_list, list_to_columnsPDF functions. Requirespdftotexton your pathdataplaybook.tasks.io_pdfread_pdf_pages, read_pdf_filesRead XMLdataplaybook.tasks.io_xmlread_xmlData Playbook v0Thev0of dataplaybook used yaml files, very similar to playbooksUse:dataplaybook playbook.yamlPlaybook structureThe playbook.yaml file allows you to load additional modules (containing tasks) and specify the tasks to execute in sequence, with all their parameters.Thetasksto perform typically follow the the structure of read, process, write.Example yaml: (please note yaml is case sensitive)modules:[list,of,modules]tasks:-task_name:# See a list of tasks belowtask_setting_1:1tables:# The INPUT. One of more tables used by this tasktarget:# The OUTPUT. Target table name of this functiondebug:True/False# Print extra debug message, default: FalseTemplatingJinja2 and JMESPath expressions can be used to create parameters for subsequent tasks. Jinja2 simply use the"{{ var[res1] }}"bracket syntax and jmespath expressions should start with the wordjmespathfollowed by a space.Both thevarsandtemplatetasks achieve a similar result: (this will search a table matching string "2" on the key column and return the value in the value column)-vars:res1:jmespath test[?key=='2'].value | [0]# is equal to-template:jmespath:"test[?key=='2'].value|[0]"target:res1# ... then use it with `{{ var.res1 }}`The JMESpath tasktemplatetask has an advantage that you can create new variablesor tables.If you have a lookup you use regularly you can do the following:-build_lookup_var:key:keycolumns:[value]target:lookup1# and then use it as follows to get a similar results to the previous example-vars:res1:"{{var['lookup1']['2'].value}}"When searching through a table with Jinja, a similar one-liner, usingselectattr, seems much more complex:-vars:res1:"{{test|selectattr('key','equalto','2')|map(attribute='value')|first}}"Special yaml functions!re <expression>Regular expression!es <search string>Search a file using Everything by VoidtoolsInstall the development versionClone the repopip install <path> -eData Playbook v0 originsData playbooks was created to replace various snippets of code I had lying around. They were all created to ensure repeatability of some menial task, and generally followed a similar structure of load something, process it and save it. (Process network data into GIS tools, network audits & reporting on router & NMS output, Extract IETF standards to complete SOCs, read my bank statements into my Excel budgeting tool, etc.)For many of these tasks I have specific processing code (tasks_x.py, loaded withmodules: [tasks_x]in the playbook), but in almost all cases input & output tasks (and configuring these names etc) are common. The idea of the modular tasks originally came from Home Assistant, where I started learning Python and the idea of "custom components" to add your own integrations, although one could argue this also has similarities to Ansible playbooks.In many cases I have a 'loose' coupling to actual file names, using Everything search (!es search_patternin the playbook) to resolve a search pattern to the correct file used for input.It has some parts in common with Ansible Playbooks, especially the name was chosen after I was introduced to Ansible Playbooks. The task structure has been updated in 2019 to match the Ansible Playbooks 2.0/2.5+ format and allow names. This format will also be easier to introduce loop mechanisms etc.Comparison to Ansible PlaybooksData playbooks is intended to create and modify variables in the environment (similar toinventory). Data playbooks starts with an empty environment (although you can read the environment from various sources inside the play).
Although new variables can be created usingregister:in Ansible, data playbook functions requires the output to be captured throughtarget:.Data playbook tasks are different form Ansible'sactions:They are mostly not idempotent, since the intention is to modify tables as we go along,they can return lists containing rows or be Python iterators (thatyieldrows of a table)if they dont return any tabular data (a list), the return value will be added to thevartable in the environmentEach have a strict voluptuous schema, evaluated when loading and during runtime (e.g. to expand templates) to allow quick troubleshootingYou could argue I can do this with Ansible, but it won't be as elegant with single item hosts files,gather_facts: noanddelegate_to: localhostthroughout the playbooks. It will likely only be half as much fun trying to force it into my way of thinking.