Package: graphreduce
GraphReduceDescriptionGraphReduce is an abstraction for building machine learning feature
engineering pipelines that involve many tables in a composable way.
The library is intended to help bridge the gap between research feature
definitions and production deployment without the overhead of a full
feature store.  Underneath the hood, GraphReduce uses graph data
structures to represent tables/files as nodes and foreign keys
as edges.Compute backends supported:pandas,dask, andspark.
Compute backends coming soon:rayInstallation# from pypi
pip install graphreduce

# from github
pip install 'graphreduce@git+https://github.com/wesmadrigal/graphreduce.git'

# install from source
git clone https://github.com/wesmadrigal/graphreduce && cd graphreduce && python setup.py installMotivationMachine learning requiresvectors of data, but our tabular datasets
are disconnected.  They can be represented as a graph, where tables
are nodes and join keys are edges.  In many model building scenarios
there isn't a nice ML-ready vector waiting for us, so we must curate
the data by joining many tables together to flatten them into a vector.
This is the problemgraphreducesets out to solve.An example dataset might look like the following:To get this example schema ready for an ML model we need to do the following:define the node-level interface and operations for filtering, annotating, normalizing, and reducingselect thegranularity) to which we'll reduce our data: in this examplecustomerspecify how much historical data will be included and what holdout period will be used (e.g., 365 days of historical data and 1 month of holdout data for labels)filter all data entities to include specified amount of history to preventdata leakagedepth first, bottom up aggregation operations group by / aggregation operations to reduce dataDefine the node-level interface and operationsimportdatetimefromgraphreduce.nodeimportGraphReduceNodefromgraphreduce.enumimportComputeLayerEnum,PeriodUnitfromgraphreduce.graph_reduceimportGraphReduce# Convention over configuration requires that we# define boilerplate code for every entity / node# we will compute over.classCustomerNode(GraphReduceNode):defdo_annotate(self):passdefdo_filters(self):# Apply a filter operation on a hypothetical column `is_fake`.# The `colabbr` method makes sure to prefix the column with# the class or instance prefix.self.df=self.df[self.df[self.colabbr('is_fake')]==False]defdo_normalize(self):passdefdo_post_join_annotate(self):passdefdo_reduce(self,reduce_key,*args,**kwargs):passdefdo_labels(self,reduce_key,*args,**kwargs):passclassOrderNode(GraphReduceNode):defdo_annotate(self):passdefdo_filters(self):passdefdo_normalize(self):passdefdo_post_join_annotate(self):passdefdo_reduce(self,reduce_key):# The `prep_for_features` method ensures no leakage# prior to the compute period or after the cut date.returnself.prep_for_features().groupby(self.colabbr(reduce_key)).agg(**{self.colabbr(f'{self.pk}_count'):pd.NamedAgg(column=self.colabbr(self.pk),aggfunc='count')}).reset_index()defdo_labels(self,key):passInstantiate the nodes and define the graphcust=CustomerNode(pk='id',prefix='cust',fpath='dat/cust.csv',fmt='csv',compute_layer=ComputeLayerEnum.pandas)order=OrderNode(pk='id',prefix='order',fpath='dat/orders.csv',fmt='csv',compute_layer=ComputeLayerEnum.pandas)gr=GraphReduce(cut_date=datetime.datetime(2023,5,6),compute_period_val=365,compute_period_unit=PeriodUnit.day,parent_node=cust,compute_layer=ComputeLayerEnum.pandas,has_labels=False,label_period_val=30,label_period_unit=PeriodUnit.day,dynamic_propagation=True)# Add nodes and edges to the graphgr.add_node(cust)gr.add_node(order)gr.add_entity_edge(parent_node=cust,relation_node=order,parent_key='id',relation_key='customer_id',relation_type='parent_child',reduce=True)Plot the graph reduce compute graph.gr.plot_graph('my_graph_reduce.html')Run compute operationsgr.do_transformations()2023-08-0309:05:44[info]hydratinggraphattributes2023-08-0309:05:44[info]hydratingattributesforCustomerNode2023-08-0309:05:44[info]hydratingattributesforOrderNode2023-08-0309:05:44[info]hydratinggraphdata2023-08-0309:05:44[info]checkingforprefixuniqueness2023-08-0309:05:44[info]runningfilters,normalize,andannotationsforCustomerNode2023-08-0309:05:44[info]runningfilters,normalize,andannotationsforOrderNode2023-08-0309:05:44[info]depth-firsttraversalthroughthegraphfromsource:CustomerNode2023-08-0309:05:44[info]reducingrelationOrderNode2023-08-0309:05:44[info]doingdynamicpropagationonnodeOrderNode2023-08-0309:05:44[info]joiningOrderNodetoCustomerNodeUse materialized dataframe for ML / analyticsgr.df.head()cust_idcust_nameorder_customer_idorder_id_countorder_id_minorder_id_maxorder_id_sumorder_customer_id_minorder_customer_id_maxorder_customer_id_sumorder_ts_first01wes121231122023-05-1212john223472242023-01-01order of operationsAPI definitionGraphReduce instantiation and parametersgraphreduce.graph_reduce.GraphReducecut_datecontrols the date around which we orient the data in the graphcompute_period_valcontrols the amount of time back in history we consider during compute over the graphcompute_period_unittells us what unit of time we're usingparent_nodespecifies the parent-most node in the graph and, typically, the granularity to which to reduce the datafromgraphreduce.graph_reduceimportGraphReducefromgraphreduce.enumsimportPeriodUnitgr=GraphReduce(cut_date=datetime.datetime(2023,2,1),compute_period_val=365,compute_period_unit=PeriodUnit.day,parent_node=customer)GraphReduce commonly used functionsdo_transformationsperform all data transformationsplot_graphplot the graphadd_entity_edgeadd an edgeadd_nodeadd a nodeNode definition and parametersgraphreduce.node.GraphReduceNodedo_annotateannotation definitions (e.g., split a string column into a new column)do_filtersfilter the data on column(s)do_normalizeclip anomalies like exceedingly large values and do normalizationpost_join_annotateannotations on current node after relations are merged in and we have access to their columns, toodo_reducethe most import node function, reduction operations: group bys, sum, min, max, etc.do_labelslabel definitions if any# alternatively can use a dynamic nodefromgraphreduce.nodeimportDynamicNodedyna=DynamicNode(fpath='s3://some.bucket/path.csv',compute_layer=ComputeLayerEnum.dask,fmt='csv',prefix='myprefix',date_key='ts',pk='id')Node commonly used functionscolabbrabbreviate a columnprep_for_featuresfilter the node's data by the cut date and the compute period for point in time correctness, also referred to as "time travel" in blogsprep_for_labelsfilter the node's data by the cut date and the label period to prepare for labelingRoadmapintegration with Raymore dynamic feature engineering abilities, possible integration with Deep Feature Synthesis