Package: adf
Table of contentsOverviewFlow configuration documentationImplementer configuration documentationAPI documentationProcessing functionsOverviewInstallIt is highly advised to install the ADF framework in a virtual env runningpython3.7, as this is (currently) the only supported python version for the AWS implementer. In addition, make sure to properly set yourPYSPARK_PYTHONpath for full spark support :mkvirtualenv adf -p `which python3.7`
export PYSPARK_PYTHON=`which python3`
pip install adfADF in a nutshellAbstract Data Flows (ADF)is a framework that provides data platform automation without infrastructure commitment. Data processing flows are defined in an infrastructure agnostic manner, and are then plugged into anyimplementerconfiguration of your choice. This provides all the major benefits of automation (namely, instantly deployable production ready infrastructure) while avoiding its major pitfall : being tied to your choice of infrastructure.Getting startedFor an easy-to-follow tutorial, please refer to the accompanyingadf_appsister repository and its associated README.Flow configuration documentationGlobal configurationModulesFlowsStarting stepsLanding stepCombination stepReception stepNon-starting stepMetadata configurationEach flow configuration file defines anADF collection. The configuration can be broken down into 3 categories of parameters.Global configurationParameterObligationDescriptionnameREQUIREDThe name for the ADF collection.BATCH_ID_COLUMN_NAMEOPTIONAL, advised not to changeColumn name to store batch IDs.SQL_PK_COLUMN_NAMEOPTIONAL, advised not to changeColumn name to store a PK if needed.TIMESTAMP_COLUMN_NAMEOPTIONAL, advised not to changeColumn name to store timestamps.For example :name:collection-nameBATCH_ID_COLUMN_NAME:modified-batch-id-column-nameSQL_PK_COLUMN_NAME:modified-sql-pk-column-nameTIMESTAMP_COLUMN_NAME:modified-timestamp-column-nameModulesModules are listed under themodulesparameter. Each module must define the following parameters :ParameterObligationDescriptionnameREQUIREDAlias to refer to this module.import_pathREQUIREDValid python import path.For example :modules:-name:module-aliasimport_path:package.moduleFlowsThe actual data flows are defined under theflowsparameter, as a list of named flows, each containing a list of named steps.ParameterObligationDescriptionnameREQUIREDUnique Identifier.stepsREQUIREDList of steps.For example :collection:collection-namemodules:-[...]-[...]flows:name:flow-namesteps:-[...]# Step config-[...]# Step config-[...]# Step configStarting stepsThe first step in a flow is known as a starting step. There are 3 types of starting steps.Landing stepThere are passive steps that define where input data is expected to be received. As such, they cannot define a processing function, metadata, or any custom flow control mechanisms.ParameterObligationDescriptionstartREQUIREDMust be set tolanding.layerREQUIREDData layer name.nameREQUIREDIdentifier unique within this flow.versionOPTIONALData version ID.funcBANNEDCannot define a processing function.func_kwargsBANNEDCannot define function keywords.metaBANNEDCannot define custom metadata.sequencerBANNEDCannot define a custom sequencer.data_loaderBANNEDCannot define a custom data loader.batch_dependencyBANNEDCannot define custom batch dependency.For example :name:collection-nameflows:-name:flow-namesteps:-start:landinglayer:layer-namename:landing-step-nameversion:source-data-version-nameCombination stepThese steps define the start of a flow that takes as input multiple previous steps.ParameterObligationDescriptionstartREQUIREDMust be set tocombination.layerREQUIREDData layer name.nameREQUIREDIdentifier unique within this flow.input_stepsREQUIREDInput steps to combine.versionOPTIONALData version ID.funcOPTIONALThe processing function.func_kwargsOPTIONALExtra kwargs to pass to the function.metaOPTIONALMetadata constraints on output.sequencerOPTIONALDefined for the step itself.data_loaderOPTIONALDefined for the step and the input steps.batch_dependencyOPTIONALDefined only for the input steps.A minimal working example :name:collection-namemodules:-name:module-aliasimport_path:package.moduleflows:-name:flow-name-0steps:-start:landinglayer:layer-namename:landing-step-name-name:flow-name-1steps:-start:landinglayer:layer-namename:landing-step-name-name:combination-flowsteps:-start:combinationlayer:layer-namename:combination-step-nameinput_steps:-flow_name:flow-name-0step_name:landing-step-flow_name:flow-name-1step_name:landing-stepfunc:# REQUIRED if there are more than one input stepsload_as:moduleparams:module:module-aliasname:processing_function_nameAn example with all optional configurations :name:collection-namemodules:-name:module-aliasimport_path:package.moduleflows:-name:flow-name-0steps:-start:landinglayer:layer-namename:landing-step-name-name:flow-name-1steps:-start:landinglayer:layer-namename:landing-step-name-name:combination-flowsteps:-start:combinationlayer:layer-namename:combination-step-nameversion:version-nameinput_steps:-flow_name:flow-name-0step_name:landing-stepdata_loader:module:module-aliasclass_name:DataLoaderClassNameparams:[...]# class init paramsbatch_dependency:module:module-aliasclass_name:BatchDependencyClassNameparams:[...]# class init params-flow_name:flow-name-1step_name:landing-stepdata_loader:module:module-aliasclass_name:DataLoaderClassNameparams:[...]# class init paramsbatch_dependency:module:module-aliasclass_name:BatchDependencyClassNameparams:[...]# class init paramsfunc:load_as:moduleparams:module:module-aliasname:processing_function_namefunc_kwargs:[...]# kwargs dictionarymeta:[...]# metadata for outputsequencer:module:module-aliasclass_name:SequencerClassNameparams:[...]# class init paramsdata_loader:module:module-aliasclass_name:DataLoaderClassNameparams:[...]# class init paramsReception stepA reception step is used when we want a processing step to output more than one data structure. When a processing step is hooked to reception steps, no data will actually be saved at the processing step itself. Instead, the reception steps will serve as storage steps. As a result, much like landing steps, they cannot define a processing function or any custom flow control mechanisms, but they can define metadata.ParameterObligationDescriptionstartREQUIREDMust be set toreception.layerREQUIREDData layer name.nameREQUIREDIdentifier unique within this flow.keyREQUIREDKeyword by which to specify this step.input_stepsREQUIREDUpstream steps to store results of.metaOPTIONALMetadata constraints on incoming data.versionBANNEDUses input step version.funcBANNEDCannot define a processing function.func_kwargsBANNEDCannot define function keywords.sequencerBANNEDCannot define a custom sequencer.data_loaderBANNEDCannot define a custom data loader.batch_dependencyBANNEDCannot define custom batch dependency.For example :name:collection-namemodules:-name:module-aliasimport_path:package.moduleflows:-name:flow-namesteps:-start:landinglayer:layer-namename:landing-step-name-layer:layer-namename:processing-stepfunc:# see below for example functionload_as:moduleparams:module:module-aliasname:multiple_outputs-name:reception-flow-0steps:-start:receptionlayer:layer-namename:reception-step-namekey:reception-key-0input_steps:-flow_name:flow-namestep_name:processing-step-name:reception-flow-1steps:-start:receptionlayer:layer-namename:reception-step-namekey:reception-key-1input_steps:-flow_name:flow-namestep_name:processing-stepHooking a processing step into a reception step changes the expected output signature of the processing function. Instead of returning a single ADS, it must now return a dictionary whose keys correspond to the reception step keys. In our case :defmultiple_outputs(ads:AbstractDataStructure,)->Dict[str,AbstractDataStructure]:return{"reception-key-0":ads[ads["col_0"]==0],"reception-key-1":ads[ads["col_0"]!=0],}Non-starting stepA non-starting step is any step that is not the first step in a flow. It can customize any and all flow control mechanisms, as well as define a processing function and define metadata.ParameterObligationDescriptionlayerREQUIREDData layer name.nameREQUIREDIdentifier unique within this flow.versionOPTIONALData version ID.funcOPTIONALThe processing function.func_kwargsOPTIONALExtra kwargs to pass to the function.metaOPTIONALMetadata constraints on output.sequencerOPTIONALDefines batch sequencing.data_loaderOPTIONALDefines data loading.batch_dependencyOPTIONALDefines batch dependency.startBANNEDMust not be set.A minimal working example :name:collection-nameflows:-name:flow-name-0steps:-start:landinglayer:layer-namename:landing-step-name-layer:layer-namename:processing-step-namefunc:# if not set, the input data is merely copiedload_as:evalparams:expr:'lambdaads:ads[ads["col_0"==0]]'An example with all optional configurations :name:collection-namemodules:-name:module-aliasimport_path:package.moduleflows:-name:flow-name-0steps:-start:landinglayer:layer-namename:landing-step-name-layer:layer-namename:processing-step-nameversion:version-namefunc:load_as:evalparams:expr:'lambdaads:ads[ads["col_0"==0]]'func_kwargs:[...]# kwargs dictionarymeta:[...]# metadata for outputsequencer:module:module-aliasclass_name:SequencerClassNameparams:[...]# class init paramsdata_loader:module:module-aliasclass_name:DataLoaderClassNameparams:[...]# class init paramsbatch_dependency:module:module-aliasclass_name:BatchDependencyClassNameparams:[...]# class init paramsMetadata configurationMetadata is configured by specifying column names, data types, and requested behavior when missing. You can also set the default missing column behavior, as well as what to do with extra columns. All metadata parameters except for the column name are optional.ParameterValuesDescriptioncolumn.nameAnyName of the columncolumn.caststr,int,float,complex,datetime,date,timedeltaData typecolumn.on_missingignore,fail,fillMissing column behaviorcolumn.fill_valAny, defaults toNoneValue to fill column if missingin_partitiontrue,falseWhether to use in partitionon_missing_defaultignore,fail,fillDefault missing column behavioron_extraignore,fail,cutWhat to do with extra columnsFor example :name:collection-nameflows:-name:flow-namesteps:-start:landinglayer:layer-namename:landing-step-layer:layer-namename:meta-stepmeta:columns:-name:essential_columncast:stron_missing:failin_partition:true-name:integer_columncast:inton_missing:fillfill_val:"FILL_VALUE"-name:weakly_defined_columnon_missing_default:ignoreon_extra:cutImplementer configuration documentationLocal ImplementerAWS ImplementerManaged infrastructure AWS ImplementerPrebuilt infrastructure AWS ImplementerWhile implementer configurations are allowed to vary freely, there is one parameter they must all contain to actually specify which implementer they are destined for. Its value must be a valid python import path.ParameterDescriptionimplementer_classModule path followed by implementer class nameFor example, if the implementer class is defined in the modulepackage.module, and the implementer class name isImplementerClass, the corresponding configuration would be :implementer_class:package.module.ImplementerClassLocal ImplementerThe local implementer requires a root path, as well as a list of layer names to associate to each layer type. The available layer types are :3 file based CSV data layers, each of which manipulates data differently to perform computation :list_of_dicts: Loads data as a list of dictionaries to perform computation.pandas: Loads data as pandas DataFrame to perform computation.spark: Loads data as a pyspark DataFrame to perform computation.2 database backed data layers, each of which uses a different database engine to perform storage and computation :sqlite: Uses an sqlite database to store and process data.postgres: Uses a postgresql database to store and process data.If at least onepostgreslayer is used, then connection information must also be passed. Admin credentials may also be passed if one wishes for the implementer to create the database and technical user in question.ParameterObligationDescriptionimplementer_classREQUIREDModule path followed by implementer class nameroot_pathREQUIREDRoot path to store data and state handlerextra_packagesOPTIONALList of local paths to any packages requiredlayers.list_of_dictsOPTIONALList of list of dict based layerslayers.pandasOPTIONALList of pandas based layerslayers.sparkOPTIONALList of pyspark based layerslayers.sqliteOPTIONALList of sqlite based layerslayers.postgresOPTIONALList of postgres based layerspostgres_configOPTIONALhost,port,db,user,pw,admin_user,admin_pwFor example, to configure a local implementer without any postgres based layers :implementer_class:ADF.components.implementers.MultiLayerLocalImplementerextra_packages:[.]root_path:path/to/datalayers:pandas:-pandas-layer-name-0-pandas-layer-name-1spark:-spark-layer-name-0-spark-layer-name-1sqlite:-sqlite-layer-name-0-sqlite-layer-name-1To be able to include postgres based layers, one must add connection information :implementer_class:ADF.components.implementers.MultiLayerLocalImplementerextra_packages:[.]root_path:path/to/datapostgres_config:db:adf_db# Requireduser:adf_user# Requiredpw:pw# Requiredhost:localhost# Optional, defaults to localhostport:5432# Optional, defaults to 5432admin_user:postgres# Optional, will be used to create db and user if neededadmin_pw:postgres# Optional, will be used to create db and user if neededlayers:pandas:-pandas-layer-name-0-pandas-layer-name-1spark:-spark-layer-name-0-spark-layer-name-1postgres:-postgres-layer-name-0-postgres-layer-name-1AWS ImplementerManaged infrastructure AWS ImplementerThe AWS implementer configuration file is similar in structure to that of the local implementer. When ADF is given free rein over infrastructure deployment, individual layers carry sizing information. In addition, the state handler configuration and sizing must also be specified.ParameterObligationDescriptionimplementer_classREQUIREDModule path followed by implementer class namemodeREQUIREDSet tomanagedto tell your implementer to handle infrastructure deploymentnameREQUIREDAn identifier for the implementerlog_folderREQUIREDA local folder in which to store subcommand logsbucketREQUIREDThe S3 bucket used for data storages3_prefixREQUIREDS3 prefix for all data and uploaded configurationstate_handlerREQUIREDengine,db_name,db_instance_class,allocated_storageextra_packagesOPTIONALList of local paths to any additional required packageslambda_layersOPTIONALsep,timeout,memoryemr_layersOPTIONALmaster_instance_type,slave_instance_type,instance_count,step_concurrency,format,landing_formatemr_serverless_layersOPTIONALinitial_driver_worker_count,initial_driver_cpu,initial_driver_memory,initial_executor_worker_count,initial_executor_cpu,initial_executor_memory,max_cpu,max_memory,idle_timeout_minutesredshift_layersOPTIONALdb_name,node_type,number_of_nodesathena_layerOPTIONALlanding_formatFor example :implementer_class:ADF.components.implementers.AWSImplementerextra_packages:[.]mode:managed# ADF will handle infrastructure deploymentname:implementer-namelog_folder:local/path/to/logsbucket:YOUR-BUCKET-NAME-HEREs3_prefix:YOUR_S3_PREFIX/state_handler:engine:postgres# only postgres is currently supporteddb_name:ADF_STATE_HANDLERdb_instance_class:db.t3.microallocated_storage:20lambda_layers:lambda-layer-name:sep:","# separator for CSVstimeout:60memory:1024emr_layers:heavy:master_instance_type:m5.xlargeslave_instance_type:m5.xlargeinstance_count:1step_concurrency:5format:parquet# the format in which to store datalanding_format:csv# the format in which to expect data in landing stepsemr_serverless_layers:serverless:initial_driver_worker_count:1initial_driver_cpu:"1vCPU"initial_driver_memory:"8GB"initial_executor_worker_count:1initial_executor_cpu:"1vCPU"initial_executor_memory:"8GB"max_cpu:"32vCPU",max_memory:"256GB",idle_timeout_minutes:15,redshift_layers:expose:db_name:exposenumber_of_nodes:1node_type:ds2.xlargeathena_layers:dump:landing_format:csv# the format in which to expect data in landing stepsPrebuilt infrastructure AWS ImplementerIf you wish to connect your AWS implementer to pre-existing infrastructure, you can do this by changing the implementer mode toprebuilt. Once the implementer setup is run, it is possible to output aprebuiltconfiguration and use it moving forward. This is the recommended usage, asprebuiltmode requires fewer permissions to run, as well as fewer API calls to determine the current state of the infrastructure. Unlike inmanagedmode, no sizing information is provided. Instead, we pass endpoints and various configurations that define the data layer.ParameterObligationDescriptionimplementer_classREQUIREDModule path followed by implementer class namemodeREQUIREDSet tomanagedto tell your implementer to handle infrastructure deploymentnameREQUIREDAn identifier for the implementerlog_folderREQUIREDA local folder in which to store subcommand logsbucketREQUIREDThe S3 bucket used for data storages3_prefixREQUIREDS3 prefix for all data and uploaded configurationstate_handler_urlREQUIREDURL to state handler DBextra_packagesOPTIONALList of local paths to any additional required packageslambda_layersOPTIONALlambda_arn,lambda_name,s3_fcp_template,s3_icp,sep,sqs_arn,sqs_name,sqs_urlemr_layersOPTIONALbucket,s3_prefix,cluster_id,cluster_arn,name,public_dns,log_uri,format,landing_formatemr_serverless_layersOPTIONALapplication_id,bucket,environ,format,landing_format,role_arn,s3_fcp_template,s3_icp,s3_launcher_key,s3_prefix,venv_package_keyredshift_layersOPTIONALtable_prefix,endpoint,port,db_name,user,role_arnathena_layersOPTIONALbucket,db_name,landing_format,s3_prefix,table_prefixFor example :implementer_class:ADF.components.implementers.AWSImplementerextra_packages:[.]mode:prebuilt# ADF will plug into pre-existing infrastructurename:implementer-namelog_folder:local/path/to/logsbucket:YOUR-BUCKET-NAME-HEREs3_prefix:YOUR_S3_PREFIX/state_handler_url:postgresql://username:password@state.handler.db.url:5432/DB_NAMElambda_layers:light:lambda_arn:LAMBDA_ARNlambda_name:LAMBDA_FUNCTION_NAMEs3_fcp_template:s3://TEMPLATE/TO/FCP/PATH/fcp.{collection_name}.yamls3_icp:s3://ICP/PATH/icp.yamlsep:','# separator for CSVssqs_arn:SQS_ARNsqs_name:SQS_QUEUE_NAMEsqs_url:https://url.to/sqs/queueemr_layers:heavy:bucket:YOUR-BUCKET-NAME-HEREs3_prefix:S3/PREFIX/# where to store data in the bucketcluster_id:EMR_CLUSTER_IDcluster_arn:EMR_CLUSTER_ARNname:EMR_CLUSTER_NAMEpublic_dns:https://url.to.emr.clusterlog_uri:s3://PATH/TO/LOGS/format:parquet# the format in which to store datalanding_format:csv# the format in which to expect data in landing stepsemr_serverless_layers:serverless:application_id:app-idbucket:YOUR-BUCKET-NAME-HEREenviron:AWS_DEFAULT_REGION:aws-regionRDS_PW:RDS_STATE_HANDLER_PASSWORDREDSHIFT_PW:REDSHIFT_PASSWORDformat:parquet# the format in which to store datalanding_format:csv# the format in which to expect data in landing stepsrole_arn:EXECUTION_ROLE_ARNs3_fcp_template:s3://TEMPLATE/TO/FCP/PATH/fcp.{collection_name}.yamls3_icp:s3://ICP/PATH/icp.yamls3_launcher_key:KEY/TO/ADF/LAUNCHER/adf-launcher.pys3_prefix:S3/PREFIX/# where to store data in the bucketvenv_package_key:S3/PREFIX/venv_package.tar.gzredshift_layers:expose:table_prefix:TABLE_PREFIXendpoint:https://url.to.dbport:PORT_NUMBERdb_name:DB_NAMEuser:DB_USERNAMErole_arn:EXECUTION_ROLE_ARNathena_layers:dump:bucket:YOUR-BUCKET-NAME-HEREdb_name:ATHENA_DB_NAMElanding_format:csv# the format in which to expect data in landing stepss3_prefix:S3/PREFIX/TO/DATA/# where to store data in the buckettable_prefix:'expose_'API documentationAbstractDataStructureColumn manipulation methodsData accessAggregation methodsAbstractDataColumnColumn operationsColumn aggregationsOperatorsAbstractDataInterfaceAbstractStateHandlerADFSequencer and ADFCombinationSequencerADFSequencerADFCombinationSequencerADFDataLoaderADFBatchDependencyHandlerAbstractDataStructureAnAbstract Data Structure(ADS) provides a dataframe like API for data manipulation. This is the native input and output format for your processing functions, barring concretization. The actual execution details of the below methods will depend on which type of ADS the underlying data layer has provided us with (Pandas based, Spark based, SQL based etc.).Column manipulation methodsdeflist_columns(self)->List[str]Lists columns currently in the ADS.defcol_exists(self,col_name:str)->boolCheck if columncol_nameexists.defprune_tech_cols(self)->"AbstractDataStructure"Removes technical columns from the ADS.defrename(self,names:Dict[str,str])->"AbstractDataStructure"Renames columns from the keys of thenamesdictionary to the values of thenamesdictionary.Data accessdef__getitem__(self,key:Union[str,AbstractDataColumn,List[str]])->Union["AbstractDataStructure",AbstractDataColumn]Ifkeyis a string, returns the corresponding column :ads["col_name"]Ifkeyis anAbstractDataColumn, return an ADS filtered based on the truth value of the column :ads[ads["col_name"] == "filter_value"]Ifkeyis a list of strings, returns an ADS containing only the subset of columns specified inkey:ads[["col_0", "col_1"]]def__setitem__(self,key:Union[str,Tuple[str,AbstractDataColumn]],value:Union[Any,AbstractDataColumn],)->NoneIfvalueis anAbstractDataColumn, use it to set the specified entries in the ADS :ads["col_name"] = ads["col_name"]*2Ifvalueis any other type, fill every specified entry with its value :ads["col_name"] = "col_value"Ifkeyis a string, set the values of the corresponding column. Creates the column if it does not already exist.Ifkeyis a(str, AbstractDataColumn)type tuple, set the values of the columnkey[0]only for rows filtered bykey[1]. Note thatkey[0]must necessarily already exist as a column. Can set using either a constant value or another column. For example:ads["col_0",ads["col_1"]=="FILTER_VAL"]="SOME_VAL"ads["col_0",ads["col_1"]=="FILTER_VAL"]=ads["col_2"]defto_list_of_dicts(self)->List[Dict]Returns a list of dictionaries, each of which corresponds to a single row in the ADS.Aggregation methodsdef__len__(self)->intReturns the number of rows in the ADS.def__bool__(self)->boolReturnFalseif the ADS has 0 rows,Trueotherwise.defjoin(self,other:"AbstractDataStructure",left_on:List[str],right_on:List[str],how:Literal["left","right","outer","inner","cross"]="inner",l_modifier:Callable[[str],str]=lambdax:x,r_modifier:Callable[[str],str]=lambdax:x,modify_on:bool=True,)->"AbstractDataStructure"Joins 2 ADS objects together.other: The right-hand ADS in the join.left_on: The left-hand columns on which to join.right_on: The right-hand columns on which to join.how: The join type.l_modifier: A function that modifies column names for the left-hand ADS.r_modifier: A function that modifies column names for the right-hand ADS.modify_on: Specify whether the column name modification functions should apply to the join columns.defgroup_by(self,keys:List[str],outputs:Dict[str,Tuple[Callable[["AbstractDataStructure"],Any],Type,],],)->"AbstractDataStructure"Performs a group by operation on a given ADS.keys: List of columns on which to group.outputs: Dictionary defining aggregations to perform. The dict key is the output column name. The dict value is a 2-tuple whose first entry is a callable defining the aggregation, and whose second entry is the output type.For example, to group on columnscol_0andcol_1, and compute the integer maximum and minimum values of columncol_2for each group, one would write :ads.group_by(keys=["col_0","col_1"],outputs={"min_col_2":(lambdaads:ads["col_2"].min(),int),"max_col_2":(lambdaads:ads["col_2"].max(),int),},)defunion(self,*others:"AbstractDataStructure",all:bool=True)->"AbstractDataStructure"Performs a union with all given input ADSs.others: A varargs list of ADSs.all: IfFalse, deduplicate results.defdistinct(self,keys:Optional[List[str]]=None)->"AbstractDataStructure"Deduplicate entries. Can optionally deduplicate only on a subset of columns by specifying thekeysarguments.defapply(self,output_column:str,func:Callable[[Dict],Any],cast:Type)->"AbstractDataStructure"Apply a User Defined Function (UDF) on the ADS.output_column: Name of the output column that will contain the result of the UDF.func: The UDF in question. Takes a dict as input that corresponds to a given row of the ADS.cast: The output data type.For example :ads.apply("output_col",lambdax:str(x["col_0"]).upper(),str)defsort(self,*cols:str,asc:Union[bool,List[bool]]=True)->"AbstractDataStructure"Sort the ADS along the given columns. Set if ascending or descending order usingasc.deflimit(self,n:int)->"AbstractDataStructure"Output a subset of the ADS based on the given number of rows.AbstractDataColumnAnAbstract Data Columnis a column of an ADS. Much like with an ADS, specific execution details vary based on the ADS it originated from (Pandas based, Spark based, SQL based etc.).Column operationsdefas_type(self,t:Type,**kwargs)->"AbstractDataColumn"Cast a column to the requested type. Acceptable types are :strintfloatcomplexbooldatetime.datetime: default conversion options may be overridden by specifying kwargsauto_convert,as_timestamp, anddatetime_format.datetime.datedatetime.timedeltadefisin(self,comp:List)->"AbstractDataColumn"Returns a boolean column where rows are set toTruewhen entries are in the givencomplist, andFalseotherwise.Column aggregationsdefmin(self)->AnyReturns minimum value of column.defmax(self)->AnyReturns maximum value of column.defsum(self)->AnyReturns sum of all column entries.defmean(self)->AnyReturns average value of column entries.defcount(self)->intdef__len__(self)->intReturns number of rows in column.def__bool__(self)->boolReturnFalseif the column has 0 rows,Trueotherwise.OperatorsAll binary and unary operators are supported. For example :ads["col_0"]*22-ads["col_0"]ads["col_0"]/ads["col_1"]~(ads["col_0"]==ads["col_1"])AbstractDataInterfaceAnAbstract Data Interfacehandles all matters related to persisting data. Abstract Data Interfaces correspond either directly to a given data layer, or to a transition between 2 data layers. Much like with an ADS, execution details depend on the underlying persistance details (file based, database based, cloud based etc.).defread_batch_data(self,step:ADFStep,batch_id:str)->AbstractDataStructureFor a given step and batch ID, return the corresponding ADS.defread_full_data(self,step:ADFStep)->AbstractDataStructureFor a given step return all available data.defread_batches_data(self,step:ADFStep,batch_ids:List[str])->Optional[AbstractDataStructure]For a given step and a list of batch IDs, return the corresponding ADS. ReturnsNoneif the input batch ID list is empty.defwrite_batch_data(self,ads:AbstractDataStructure,step:ADFStep,batch_id:str)->NoneGiven an ADS and a target step and batch ID, persist the ADS.defdelete_step(self,step:ADFStep)->NoneDelete all data in the given step.defdelete_batch(self,step:ADFStep,batch_id:str)->NoneDelete data corresponding to a specific batch ID for a given step.AbstractStateHandlerAnAbstract State Handlercontains all information related to the current processing state. In particular, it can list all batch IDs and their current state.defto_ads(self)->AbstractDataStructureReturns an ADS describing all batches. The output ADS will always have the following columns :collection_nameflow_namestep_nameversionlayerbatch_idstatusdatetimemsgThis method gives you complete read capabilities on the state handler, allowing you to extract any information you need from it. All following methods are merely shortcuts built on top of this one.defto_step_ads(self,step:ADFStep)->AbstractDataStructureReturns an ADS containing the processing state of a given ADF step.defget_entries(self,collection_name:Optional[str]=None,flow_name:Optional[str]=None,step_name:Optional[str]=None,version:Optional[str]=None,layer:Optional[str]=None,batch_id:Optional[str]=None,status:Optional[str]=None,)->List[Dict]Returns a list of batch IDs corresponding to the given filters.defget_step_submitted(self,step:ADFStep)->List[str]defget_step_running(self,step:ADFStep)->List[str]defget_step_deleting(self,step:ADFStep)->List[str]defget_step_failed(self,step:ADFStep)->List[str]defget_step_success(self,step:ADFStep)->List[str]For a given ADF step, return all batch IDs in the given state (submitted,running,deleting,failed, orsuccess).defget_step_all(self,step)->List[str]For a given ADF step, return all batch IDs.defget_batch_info(self,step:ADFStep,batch_id:str)->DictFor a given batch ID of a given ADF step, return a dictionary containing all batch information.defget_batch_status(self,step:ADFStep,batch_id:str)->strFor a given ADF step, returns the status of a given batch ID. Raises an error if the batch ID is unknown to the state handler.ADFSequencer and ADFCombinationSequencerAnADF Sequencerdefines the batches to be processed by a given step at any given time. To define your own ADF Sequencer, you must inherit from either theADFSequencerorADFCombinationSequencerbase class (the latter should only be used for combination steps). In both cases, there are 2 abstract methods that require defining.ADFSequencerdeffrom_config(cls,config:Dict)->"ADFSequencer"Given configuration parameters, return anADFSequencerinstance. If you define a custom__init__for this, make sure it callssuper().__init__().defget_to_process(self,state_handler:AbstractStateHandler,step_in:ADFStep,step_out:ADFStep,)->List[str]Input arguments :state_handler: Contains the current processing state.step_in: The input step.step_out: The output step.How the output shapes the flow of data :The return value is the list of batch IDs the output step is expected to create.By default, previously submitted batches are ignored, there is no need for your method to check for them.If you want your sequencer to resubmit such batches, you have to explicitly setredotoTruein your constructor by callingsuper().__init__(redo=True).ADFCombinationSequencerdeffrom_config(cls,config:Dict)->"ADFCombinationSequencer"Given configuration parameters, return anADFCombinationSequencerinstance. If you define a custom__init__for this, make sure it callssuper().__init__().defget_to_process(self,state_handler:AbstractStateHandler,combination_step:ADFCombinationStep,)->List[Tuple[List[str],str]]Input arguments :state_handler: Contains the current processing state.combination_step: The combination step in question.How the output shapes the flow of data :The return value is a list of 2-tuples :The first entry is a list of batch IDs corresponding to the input steps.The second entry is the corresponding batch ID output by the combination step.By default, previously submitted batches are ignored, there is no need for your method to check for them.If you want your sequencer to resubmit such batches, you have to explicitly setredotoTruein your constructor by callingsuper().__init__(redo=True).ADFDataLoaderAnADF Data Loaderdefines what data is passed to your processing function at each step. To define your own ADF Data Loader, you must inherit from theADFDataLoaderbase class. There are 2 abstract methods that then require defining.deffrom_config(cls,config:Dict)->"ADFDataLoader"Given configuration parameters, return anADFDataLoaderinstance. If you define a custom__init__for this, make sure it callssuper().__init__().defget_ads_args(self,data_interface:AbstractDataInterface,step_in:ADFStep,step_out:ADFStep,batch_id:str,state_handler:AbstractStateHandler,)->Tuple[List[AbstractDataStructure],Dict[str,AbstractDataStructure],]Input arguments :data_interface: The data persistance interface from which to load data.step_in: The input step.step_out: The output step.batch_id: The output batch ID.state_handler: Contains the current processing state.How the output is passed to your data processing functions :The first output is a list of ADSs. These are passed as positional arguments.The second output is a dict, for which each entry is passed as a keyword argument.Simply put, if yourget_ads_argsmethod returnsargs, kwargs, then these are passed to your processing functionfuncsimply as :func(*args,**kwargs)ADFBatchDependencyHandlerAnADF Batch Dependency Handlerdefines which batches are defined as beingdownstreamof batches in previous steps. This is mainly used to define which downstream batches are also deleted when deleting a particular batch of data. To define your own ADF Batch Dependency Handler, you must inherit from theADFBatchDependencyHandlerbase class. There are 2 abstract methods that then require defining.deffrom_config(cls,config:Dict)->"ADFBatchDependencyHandler"Given configuration parameters, return anADFBatchDependencyHandlerinstance. If you define a custom__init__for this, make sure it callssuper().__init__().defget_dependent_batches(self,state_handler:AbstractStateHandler,step_in:ADFStep,step_out:ADFStep,batch_id:str,)->List[str]Input arguments :state_handler: Contains the current processing state.step_in: The input step.step_out: The output step.batch_id: The input batch ID.The return value represents the list of batch IDs in the output step that are considered dependent on the given batch ID in the input step.Processing functionsInput signatureNon-starting stepCombination stepOutput signatureConcretizationThe processing function signature depends on the specific step configuration. In particular,data loaderswill modify the expected input arguments, and the presence of downstream reception steps will modify the expected output.Input signatureNon-starting stepFor non-starting steps, the input arguments will depend on the outputargsandkwargsof the step data loader, in addition to anyfunc_kwargsdefined in the step configuration. For example, consider the builtinKwargDataLoader, that merely passes the current batch of data as a keyword argument, meaningargsis an empty list andkwargsis a single entry dictionary whose key is user defined :name:collection-namemodules:-name:flow_configimport_path:ADF.components.flow_config-name:module-aliasimport_path:package.moduleflows:-name:flow-name-0steps:-start:landinglayer:layer-namename:landing-step-name-layer:layer-namename:processing-step-nameversion:version-namefunc:load_as:moduleparams:module:module-aliasname:foofunc_kwargs:extra_kwarg:kwarg_valdata_loader:module:flow_configclass_name:KwargDataLoaderparams:kwarg_name:custom_kwarg_nameIn this case, the expected function input signature is :deffoo(custom_kwarg_name:AbstractDataStructure,extra_kwarg:str)Thecustom_kwarg_nameargument will contain the ADS passed by our data loader, and theextra_kwargargument will contain the valuekwarg_valas defined in our flow configuration.Combination stepFor a combination step, theargsandkwargsare the combination of the outputs of the data loaders of all input steps, plus the data loader of the combination step itself. Again, let's use theKwargDataLoaderto illustrate this, as well as theFullDataLoaderwhich loads all available data for a given step.name:collection-namemodules:-name:flow_configimport_path:ADF.components.flow_config-name:module-aliasimport_path:package.moduleflows:-name:flow-name-0steps:-start:landinglayer:layer-namename:landing-step-name-name:flow-name-1steps:-start:landinglayer:layer-namename:landing-step-name-name:combination-flowsteps:-start:combinationlayer:layer-namename:combination-step-nameversion:version-nameinput_steps:-flow_name:flow-name-0step_name:landing-stepdata_loader:module:flow_configclass_name:KwargDataLoaderparams:kwarg_name:custom_kwarg_name_0-flow_name:flow-name-1step_name:landing-stepdata_loader:module:flow_configclass_name:KwargDataLoaderparams:kwarg_name:custom_kwarg_name_0func:load_as:moduleparams:module:module-aliasname:foofunc_kwargs:extra_kwarg:kwarg_valdata_loader:module:flow_configclass_name:FullDataLoaderparams:{}Each input step data loader will add an entry to the inputkwargs. The data loader of the combination step itself will load the full data of that same step as the sole entry of our outputargs. Finally, there is also the user definedextra_kwarg, defined directly in our flow configuration, that will enrich the inputkwargs. Putting all of this together, we get the following input signature :deffoo(full_data:AbstractDataStructure,custom_kwarg_name_0:AbstractDataStructure,custom_kwarg_name_1:AbstractDataStructure,extra_kwarg:str,)Output signatureBy default, a processing function must output a single ADS :deffoo(ads:AbstractDataStructure)->AbstractDataStructure:returnads[ads["some_col"]=="some_val"]However, hooking a processing step into a reception step changes the expected output signature of the processing function. Instead of returning a single ADS, it must now return a dictionary whose keys correspond to the reception step keys. Take the following flow configuration as an example :name:collection-namemodules:-name:module-aliasimport_path:package.moduleflows:-name:flow-namesteps:-start:landinglayer:layer-namename:landing-step-name-layer:layer-namename:processing-stepfunc:load_as:moduleparams:module:module-aliasname:multiple_outputs-name:reception-flow-0steps:-start:receptionlayer:layer-namename:reception-step-namekey:reception-key-0input_steps:-flow_name:flow-namestep_name:processing-step-name:reception-flow-1steps:-start:receptionlayer:layer-namename:reception-step-namekey:reception-key-1input_steps:-flow_name:flow-namestep_name:processing-stepA valid corresponding processing function would then be :defmultiple_outputs(ads:AbstractDataStructure,)->Dict[str,AbstractDataStructure]:return{"reception-key-0":ads[ads["col_0"]==0],"reception-key-1":ads[ads["col_0"]!=0],}ConcretizationIt is possible to define processing functions using familiar "concrete" APIs (such as Pandas, PySpark, raw SQL etc.) using a procedure known asconcretization. However, this comes at the cost that when the corresponding step is mapped to a layer, that layer must support the chosen concretization. For example, concretizing to a PySpark dataframe will fail for an SQL based layer. Our data flows remain abstract, but they become somewhat constrained in their eventual layer mapping.To define a processing function as concrete, you may use theconcretizedecorator, which takes as input a concretization type. The decorator will transform all input ADSs into the requested type. It will do so in a nested manner, also transforming ADSs within input lists, tuples, or dictionaries. It also expects that type as the function output type. :frompandasimportDataFramefromADF.utilsimportconcretize@concretize(DataFrame)deffoo(df:DataFrame)->DataFrame:returndf.drop_duplicates()There are 2 main use cases for concretization that may be worth the slight trade-off of layer constraint :Migrating a non ADF pipeline to ADF, allowing reuse of business logic code.Exploiting API specific optimizations, such asbroadcastfor a PySpark dataframe.