Package: boilerdata
boilerdataData processing pipeline for a nucleate pool boiling apparatus.OverviewThe data processing approach taken in this repository started over atpdpipewrench. It was initially conceptualized as a way to outfitpdpipepipelines from configuration files, allowing for Pandas pipeline orchestration with minimal code. I have since adopted a less aggressive tact, where I still separate configuration out into YAML files (constants, file paths, pipeline function arguments, etc.), but pipeline logic is handled inpipeline.py. I have also done away with usingpdpipein this approach, as it doesn't lend itself particularly well toETL. Besides, my data processing need is not quite the "flavor" of statistical data science type approaches supported bypdpipe.This new approach maintains the benefits of writing logic in Python, while allowing configuration in files. I am usingPydanticas the interface between my configs and my logic, which allows me to specify allowable values withEnumsand other typing constructs. Expressing allowable configurations with Pydantic allows for generation of schema for your config files, raising errors on typos or missing keys, for example. I also specify the "shape" of my input and output data in configs, and validate my dataframes withpandera. Once these components are in place, it is easy to implement new functionality in the pipeline.UsageIf you would like to adopt this approach to processing your own data, you may clone this repository and begin swapping configs and logic for your own, or use a similar architecture for your data processing. To run a working example with some actual data from this study, perform the following steps:Clone this repository and open it in your terminal or IDE (e.g.git clone https://github.com/blakeNaccarato/boilerdata.git boilerdata).Navigate to the clone directory in a terminal window (e.g.cd boilerdata).Create a Python 3.10 virtual environment (e.g.py -3.10 -m venv .venvon Windows w/ Python 3.10 installed frompython.org).Activate the virtual environment (e.g..venv/scripts/activateon Windows).Runpip install --editable .to installboilerdatapackage in an editable fashion. This step may take awhile.Delete the top-leveldataandconfigdirectories, then copy theconfiganddatafolders inside oftests/datato the root directory.Copy the.propshopfolder intests/data/.propshopto your user-folder (e.g.C:/Users/<you>/.propshopon Windows).Rundvc repro metricsto execute the data process up to that stage.The data process should run the following stages:axes,modelfun,runs,parse_benchmarsk,pipeline, andmetrics. Some stages are skipped because we specified to run just the necessary stages up tometrics(the example data doesn't currently include the literature data). You may inspect the pipeline stages of the same name insrc/boilerdata/stages, such aspipeline.pyto see the logic that runs during that stage. This example happens to use Python scripts, but you could define a stage indvc.yamlthat instead runs Matlab scripts, or any arbitrary action. This approach allows for the data process to be reliably reproduced over time, and for the process to be easily modified and extended in a collabroative effort.There are other details of this process, such as the hosting of data in thedatafolder in a Google Cloud Bucket (alternatively it can be hosted on Google Drive), and more. This has to do with the need to store data (especially large datasets) outside of the repository, and access it in an authenticated fashion.Data process graphThis data process graph is derived from the structure of the code itself. It is automatically generated bydvc. This self-documenting process improves reproducibility and reduces documentation overhead.flowchart TD
 node1["axes"]
 node2["data\benchmarks.dvc"]
 node3["data\curves.dvc"]
 node4["data\literature.dvc"]
 node5["data\plotter.dvc"]
 node6["literature"]
 node7["metrics"]
 node8["modelfun"]
 node9["originlab"]
 node10["parse_benchmarks"]
 node11["pipeline"]
 node12["runs"]
 node1-->node8
 node1-->node10
 node1-->node12
 node2-->node10
 node3-->node12
 node4-->node6
 node5-->node9
 node6-->node9
 node8-->node11
 node10-->node11
 node11-->node7
 node11-->node9
 node12-->node11