Package: ecopipeline
DataPipelinePackageTo Install the PackageFrom the internet for use elsewhere:
$ pip install ecopipeline
Install locally in an editable mode:
Navigate to DataPipelinePackage directory and run the following command
$ pip install -e .File Structure.
├── src
├   ├── docs 
├   ├   ├── 
├   └── ecopipeline
├       ├── extract.py               # functionality for extracting data from a file system
├       ├── transform.py             # functionality for cleaning data and calcualting derived values
├       ├── load.py                  # functionality for loading pandas dataframe into a mySQL database table
├       ├── unit_convert.py   
├       ├── config.py                # file containing all file paths 
├       ├── bayview.py               # Bayview site-specific functionality
├       └── lbnl.py                  # LBNL site-specific functionality
├── testing
├   ├── Bayview
├   ├    ├── Bayview_input
├   ├    ├── extract.py              
├   ├    ├   └── extract_test.py     # testing for extract functionality
├   ├    ├── transform.py
├   ├    ├   ├── pickles             # pickles used for bayview unit testing
├   ├    ├   └── transform_test.py   # testing for transform functionality
├   ├    └── load.py                 
├   ├        └── load_test.py        # testing for load functionality
├   └── LBNL
├       ├── extract.py
├       ├   └── extract_test.py
├       ├── transform.py 
├       ├   ├── LBNL-input           # LBNL input dataframes used as testing input
├       ├   ├── LBNL-output          # LBNL output dataframes used for crossreferencing our output to expected output
├       ├   ├── pickles              # pickles used for bayview unit testing
├       ├   └── transform_test.py
├       └── load.py
├           └── load_test.py
├── config.ini                       # file containing all configuration parameters
└── README.mdPurposeThis project was developed with the help of Ecotope, Inc. It containes seperate modular functionalities that, when combined, can extract, transfrom, and load data from incoming sensors. The main goal was to rewrite the existing R pipeline code with Python making the codebase more readable. In addition to that, scalability was taken into account during this project since this codebase will be used to create pipelines for different sites in the future.Architectureextract.pyloading data from a local file systemextracting NOAA weather data from a FTP servertransform.pycleaning the dataroundingremoving outliersrenaming columnsfilling missing valuescalculating dervived COP (coefficient of performance) valuesagreggating the dataload.pyestablishing a connection to the databaseloading pandas dataframe into a table in the databaseconfig.inidatabaseuser: username for host database connectionpassword: password for host database connectionhost: name of hostdatabase: name of databaseminutetable_name: name of table to be created in the mySQL database containing minute-by-minute datahourtable_name: name of table to be created in the mySQL database containing hour-by-hour datadaytable_name: name of table to be created in the mySQL database containing day-by-day datainputdirectory: diretory of the folder containing the input files listed belowsite_info: name of the site information csv410a_info: name of the 410a information csvsuperheat_info: name of the superheat infomation csvoutputdirectory: diretory of the folder where any pipeline output should be written todatadirectory: diretory of the folder from which extract loads the raw sensor dataUnit TestingTo run Unit tests, run the following command in the terminal in the corresponding directory:python-mpytest