Package: tdprepview
tdprepviewPython Package that creates Data Preparation Pipelines in Views written in Teradata-SQL.Installationpip install tdprepviewFeaturesPipeline class that allows creating in-DB preprocessing pipelinesSeveral Preprocessor functionsAPI similar to sklearn.PipelineQuickstart1. setting up the pipelineimportteradatamlastdmldatabase_credentials={# ...}DbCon=tdml.create_context(**database_credentials)myschema,table_train,table_score="...","...","..."DF_train=tdml.DataFrame(tdml.in_schema(myschema,table_train))DF_score=tdml.DataFrame(tdml.in_schema(myschema,table_score))fromtdprepviewimportPipelinefromtdprepviewimport(StandardScaler,IterativeImputer,PCA,DecisionTreeBinning)steps=[({"pattern":"^floatfeat_[1-5]$"},[StandardScaler(),IterativeImputer()],{"suffix":"_imputed"}),({"suffix":"_imputed"},PCA(n_components=2),{"prefix":"mypca_"}),({"prefix":"mypca_"},DecisionTreeBinning(target_var="target_class",no_bins=3))]mypipeline=Pipeline(steps=steps)2.a transform DataFrames directlythis is suitable if you are working in a sandbox Jupyter Notebook# fit() calculates all necessary statistics in DBmypipeline.fit(DF=DF_train)# transform generates SQL syntax based on the chosen Preprocessors and the statistics from fit()DF_train_transf=mypipeline.transform(DF_train)DF_score_transf=mypipeline.transform(DF_score)#... go on with some modelling (e.g. sklearn, TD_DecisionForest,...)# and scoring (via BYOM, TD_DecisionForestPredict, ...)2.b inspect produced queriesif you want to check the SQL code, that is generated py tdprepviewmypipeline.fit(DF=DF_train)query_train=mypipeline.transform(DF_train,return_type="str")print(query_train)the output: (Note how columns not part of the pipeline are simply forwarded)WITHpreprocessing_stepsAS(SELECTrow_idASc_i_35,floatfeat_1ASc_i_36,floatfeat_2ASc_i_37,floatfeat_3ASc_i_38,target_classASc_i_39,ZEROIFNULL(((c_i_36)--0.2331302)/NULLIF(1.747159,0))ASc_i_40,ZEROIFNULL(((c_i_37)-0.03895576)/NULLIF(1.722347,0))ASc_i_41,ZEROIFNULL(((c_i_38)-0.2363556)/NULLIF(2.808312,0))ASc_i_42,-1.118451e-08+(-0.07714142)*(COALESCE(c_i_41,1.610355e-09))+(-0.1758817)*(COALESCE(c_i_42,-4.838372e-09))ASc_i_43,4.261288e-09+(-0.0431946)*(COALESCE(c_i_40,-1.045776e-08))+(0.6412595)*(COALESCE(c_i_42,-4.838372e-09))ASc_i_44,-7.079888e-09+(-0.118112)*(COALESCE(c_i_40,-1.045776e-08))+(0.624912)*(COALESCE(c_i_41,1.610355e-09))ASc_i_45,(0.2604757)*(c_i_43)+(-0.681657)*(c_i_44)+(-0.6837369)*(c_i_45)ASc_i_46,(-0.1047098)*(c_i_43)+(0.6840609)*(c_i_44)+(-0.7218702)*(c_i_45)ASc_i_47,CASEWHENc_i_46<-2.0THEN0WHENc_i_46<-1.236351THEN1WHENc_i_46<-1.182989THEN2ELSE3ENDASc_i_48,CASEWHENc_i_47<-2.0THEN0WHENc_i_47<-0.3139175THEN1WHENc_i_47<0.2286314THEN2ELSE3ENDASc_i_49FROM<input_schema>.<input_table_view>t)SELECTc_i_35ASrow_id,c_i_48ASmypca_pc_1,c_i_49ASmypca_pc_2,c_i_39AStarget_classFROMpreprocessing_stepstif you want to inspect the pipeline as a chartmypipeline.plot_sankey()Output:2.c persist transformed data as a view for later usethis is suitable and compliant with the Teradata ModelOps-Framework, where training.py and scoring.py
are separate scripts.intraining.py:view_train_transf=table_train+"_transf_v"view_score_transf=table_score+"_transf_v"mypipeline.fit(schema_name=myschema,table_name=table_train)# 3. transform: create views for pipelines# DF_train_transf is already based on the newly created ViewDF_train_transf=mypipeline.transform(schema_name=myschema,table_name=table_train,# this triggeres the creation of a VIEW, thus the transformation# pipeline is persitedcreate_replace_view=True,output_schema_name=myschema,output_view_name=view_train_transf)# 3.b create view for scoring table - no need for further inspection, thus no returnmypipeline.transform(schema_name=myschema,table_name=table_score,return_type=None,create_replace_view=True,output_schema_name=myschema,output_view_name=view_score_transf)# further steps:# Model training with DF_train_transf,# e.g. local and use BYOM, or in-DB with TD_DecisionForest# save model in DBinscoring.py:view_score_transf=table_score+"_transf_v"# 1. get DataFrame based on View, transform is happening as per view definitionDF_score_transf=tdml.DataFrame(tdml.in_schema(myschema,view_score_transf))# 2. Model Scoring with trained model# (e.g. PMMLPredict, TD_DecisionForestPredict,...)# get pointer for model from DB + execute scoring in DB# 3. Save Scored Data in DBHistoryv0.1.0 (2023-02-15)addedFirst release on PyPI.Pipeline with fit and transform functionsPreprocessor FunctionsImputeImputeTextTryCastScaleCutOffFixedWidthBinningThresholdBinarizerListBinarizerVariableWidthBinningLabelEncoderCustomTransformerNotebooks for testsDemo Notebookv0.1.2 (2023-02-15)fixedadded *.sql to MANIFEST.ln such that SQL templates are also part of the distribution.changedHISTORY and README file from rst to Markdownv0.1.3 (2023-02-16)addedQuickstart in README filev0.1.4 (2023-02-17)addedDecisionTreeBinning as Preprocessing functionv1.0.2 (2023-03-06)Major Overhaul of tdprepview. It now supports transformations that change the schema of the Input DataFrame, like OneHotEncoding. This implementation is based on a directed acyclic graph (DAG).addedplot_sankey() Function for Pipeliine class, which plots the DAG as sankey chart.Preprocessing functionsSimpleImputer (adapted from sklearn, based on Impute)IterativeImputer (adapted from sklearn)StandardScaler (adapted from sklearn, based on Scale)MaxAbsScaler (adapted from sklearn, based on Scale)MinMaxScaler (adapted from sklearn, based on Scale)RobustScaler (adapted from sklearn, based on Scale)Normalizer (adapted from sklearn)QuantileTransformer (adapted from sklearn, based on VariableWidthBinning)Binarizer (adapted from sklearn, based on ThresholdBinarizer)PolynomialFeatures (adapted from sklearn)OneHotEncoder (adapted from sklearn)PCA (adapted from sklearn)changedexctly one query is generated, with one WITH AS part, that contains all preprocessing, one final SELECT, that contains all final columns with the correct column names.the steps in the steps argument of Pipeline are more flexible. They are either tuples with two elements (input_column(s), preprocessor(s)) or tuples with three elements (input_column(s), preprocessor(s), options)input_column(s): string, list of strings orNEWdictionary for dynamic filtering:{'pattern':<regex pattern to search for column names>}{'prefix':<string column names start with>}{'suffix':<string column names end with>}{'dtype_include':<list of tdtypes to include>}{'dtype_exclude':<list of tdtypes to exclude>}filter conditions can be combinedpreprocessor(s): one or list of Preprocessors.options: dict for renaming column names after applying preprocessor. Useful for filtering them in a later step{'prefix':<adds string to beginning of column names>}{'suffix':<adds string to end of column names>}