Package: greenguard
An open source project from Data to AI Lab at MIT.AutoML for Renewable Energy Industries.GreenGuardLicense:MITDocumentation:https://signals-dev.github.io/GreenGuardHomepage:https://github.com/signals-dev/GreenGuardOverviewThe GreenGuard project is a collection of end-to-end solutions for machine learning problems
commonly found in monitoring wind energy production systems. Most tasks utilize sensor data
emanating from monitoring systems. We utilize the foundational innovations developed for
automation of machine Learning at Data to AI Lab at MIT.The salient aspects of this customized project are:A set of ready to use, well tested pipelines for different machine learning tasks. These are
vetted through testing across multiple publicly available datasets for the same task.An easy interface to specify the task, pipeline, and generate results and summarize them.A production ready, deployable pipeline.An easy interface totunepipelines using Bayesian Tuning and Bandits library.A community oriented infrastructure to incorporate new pipelines.A robust continuous integration and testing infrastructure.Alearning databaserecording all past outcomes --> tasks, pipelines, outcomes.ResourcesData Format.GreenGuard folder structure.InstallRequirementsGreenGuardhas been developed and runs on Python 3.6, 3.7 and 3.8.Also, although it is not strictly required, the usage of avirtualenvis highly recommended in order to avoid interfering
with other software installed in the system where you are trying to runGreenGuard.Download and InstallGreenGuardcan be installed locally usingpipwith
the following command:pipinstallgreenguardThis will pull and install the latest stable release fromPyPi.If you want to install from source or contribute to the project please read theContributing Guide.Docker usageGreenGuardis prepared to be run inside a docker environment. Please check thedocker documentationfor details about how to runGreenGuardusing docker.Data FormatThe minimum input expected by theGreenGuardsystem consists of the following two elements,
which need to be passed aspandas.DataFrameobjects:Target TimesA table containing the specification of the problem that we are solving, which has three
columns:turbine_id: Unique identifier of the turbine which this label corresponds to.cutoff_time: Time associated with this targettarget: The value that we want to predict. This can either be a numerical value or a
categorical label. This column can also be skipped when preparing data that will be used
only to make predictions and not to fit any pipeline.turbine_idcutoff_timetarget0T12001-01-02 00:00:0001T12001-01-03 00:00:0012T22001-01-04 00:00:000ReadingsA table containing the signal data from the different sensors, with the following columns:turbine_id: Unique identifier of the turbine which this reading comes from.signal_id: Unique identifier of the signal which this reading comes from.timestamp (datetime): Time where the reading took place, as a datetime.value (float): Numeric value of this reading.turbine_idsignal_idtimestampvalue0T1S12001-01-01 00:00:0011T1S12001-01-01 12:00:0022T1S12001-01-02 00:00:0033T1S12001-01-02 12:00:0044T1S12001-01-03 00:00:0055T1S12001-01-03 12:00:0066T1S22001-01-01 00:00:0077T1S22001-01-01 12:00:0088T1S22001-01-02 00:00:0099T1S22001-01-02 12:00:001010T1S22001-01-03 00:00:001111T1S22001-01-03 12:00:0012TurbinesOptionally, a third table can be added containing metadata about the turbines.
The only requirement for this table is to have aturbine_idfield, and it can have
an arbitraty number of additional fields.turbine_idmanufacturer.........0T1Siemens.........1T2Siemens.........CSV FormatA part from the in-memory data format explained above, which is limited by the memory
allocation capabilities of the system where it is run,GreenGuardis also prepared to
load and work with data stored as a collection of CSV files, drastically increasing the amount
of data which it can work with. Further details about this format can be found in theproject documentation site.QuickstartIn this example we will load some demo data and classify it using aGreenGuard Pipeline.1. Load and split the demo dataThe first step is to load the demo data.For this, we will import and call thegreenguard.demo.load_demofunction without any arguments:fromgreenguard.demoimportload_demotarget_times,readings=load_demo()The returned objects are:target_times: Apandas.DataFramewith thetarget_timestable data:turbine_id cutoff_time  target
0       T001  2013-01-12       0
1       T001  2013-01-13       0
2       T001  2013-01-14       0
3       T001  2013-01-15       1
4       T001  2013-01-16       0readings: Apandas.DataFramecontaining the time series data in the format explained above.turbine_id signal_id  timestamp  value
0       T001       S01 2013-01-10  323.0
1       T001       S02 2013-01-10  320.0
2       T001       S03 2013-01-10  284.0
3       T001       S04 2013-01-10  348.0
4       T001       S05 2013-01-10  273.0Once we have loaded thetarget_timesand before proceeding to training any Machine Learning
Pipeline, we will have split them in 2 partitions for training and testing.In this case, we will split them using thetrain_test_split function from scikit-learn,
but it can be done with any other suitable tool.fromsklearn.model_selectionimporttrain_test_splittrain,test=train_test_split(target_times,test_size=0.25,random_state=0)Notice how we are only splitting thetarget_timesdata and not thereadings.
This is because the pipelines will later on take care of selecting the parts of thereadingstable needed for the training based on the information found inside
thetrainandtestinputs.Additionally, if we want to calculate a goodness-of-fit score later on, we can separate the
testing target values from thetesttable by popping them from it:test_targets=test.pop('target')2. Exploring the available PipelinesOnce we have the data ready, we need to find a suitable pipeline.The list of available GreenGuard Pipelines can be obtained using thegreenguard.get_pipelinesfunction.fromgreenguardimportget_pipelinespipelines=get_pipelines()The returnedpipelinevariable will belistcontaining the names of all the pipelines
available in the GreenGuard system:['classes.unstack_double_lstm_timeseries_classifier',
 'classes.unstack_lstm_timeseries_classifier',
 'classes.unstack_normalize_dfs_xgb_classifier',
 'classes.unstack_dfs_xgb_classifier',
 'classes.normalize_dfs_xgb_classifier']For the rest of this tutorial, we will select and use the pipelineclasses.normalize_dfs_xgb_classifieras our template.pipeline_name='classes.normalize_dfs_xgb_classifier'3. Fitting the PipelineOnce we have loaded the data and selected the pipeline that we will use, we have to
fit it.For this, we will create an instance of aGreenGuardPipelineobject passing the name
of the pipeline that we want to use:fromgreenguard.pipelineimportGreenGuardPipelinepipeline=GreenGuardPipeline(pipeline_name)And then we can directly fit it to our data by calling itsfitmethod and passing in the
trainingtarget_timesand the completereadingstable:pipeline.fit(train,readings)4. Make predictionsAfter fitting the pipeline, we are ready to make predictions on new data by calling thepipeline.predictmethod passing the testingtarget_timesand, again, the completereadingstable.predictions=pipeline.predict(test,readings)5. Evaluate the goodness-of-fitFinally, after making predictions we can evaluate how good the prediction was
using any suitable metric.fromsklearn.metricsimportf1_scoref1_score(test_targets,predictions)What's next?For more details aboutGreenGuardand all its possibilities and features, please check theproject documentation siteAlso do not forget to have a look at thetutorials!History0.3.0 - 2021-01-22This release increases the supported version of python to3.8and also includes changes
in the installation requirements, wherepandasandscikit-optimizepackages have
been updated to support higher versions. This changes come together with the newer versions
ofMLBlocksandMLPrimitives.Internal ImprovementsFixrun_benchmarkgenerating properly theinit_hyperparametersfor the pipelines.NewFPRmetric.Newroc_auc_scoremetric.Multiple benchmarking metrics allowed.Multipletprorthresholdvalues allowed for the benchmark.0.2.6 - 2020-10-23Fixmkdirwhen exporting tocsvfile the benchmark results.Intermediate steps for the pipelines with demo notebooks for each pipeline.Resolved IssuesIssue #50: Expose partial outputs and executions in theGreenGuardPipeline.0.2.5 - 2020-10-09With this release we include:run_benchmark: A function within the modulebenchmarkthat allows the user to evaluate
templates against problems with different window size and resample rules.summarize_results: A function that given acsvfile generates axlsxfile with a summary
tab and a detailed tab with the results fromrun_benchmark.0.2.4 - 2020-09-25Fix dependency errors0.2.3 - 2020-08-10Added benchmarking module.0.2.2 - 2020-07-10Internal ImprovementsAdded github actions.Resolved IssuesIssue #27: Cache Splits pre-processed data on disk0.2.1 - 2020-06-16With this release we give the possibility to the user to specify more than one template when
creating a GreenGuardPipeline. When thetunemethod of this is called, an instance of BTBSession
is returned and it is in charge of selecting the templates and tuning their hyperparameters until
achieving the best pipeline.Internal ImprovementsResample by filename inside theCSVLoaderto avoid oversampling of data that will not be used.Select targets now allows them to be equal.Fixed the csv filename format.Upgraded to BTB.Bug FixesIssue #33: Wrong default datetime formatResolved IssuesIssue #35: Select targets is too strictIssue #36: resample by filename inside csvloaderIssue #39: Upgrade BTBIssue #41: Fix CSV filename format0.2.0 - 2020-02-14First stable release:efficient data loading and preprocessinginitial collection of dfs and lstm based pipelinesoptimized pipeline tuningdocumentation and tutorials0.1.0First release on PyPI