Package: tdstone2
tdstone2PackageOverviewThetdstone2package is designed to simplify the operationalization of Python code for data analysis and machine learning on the Teradata Vantage system. It leverages the massive parallel processing architecture of Teradata Vantage to run hundreds of Python scripts across hundreds of data partitions. This approach enables the industrialization, lineage, and reproducibility of millions of custom models while minimizing data movement.FeaturesHyper-segmented Model Deployment: Deploy scikit-learn pipelines or custom Python functions across segmented datasets for parallel execution.Model Lineage and Reproducibility: Automatically track the lineage of models and ensure reproducibility across different data partitions.Efficient Data Handling: Minimize data movement by leveraging Teradata's parallel processing capabilities to execute models directly on the database.InstallationTo installtdstone2, use pip:pipinstalltdstone2Ensure you have access to a Teradata Vantage system and the necessary credentials to connect and execute queries.UsageHyper-segmented Model Deployment3.1 Engineering of the Scikit-learn Classifier Pipelinefromsklearn.preprocessingimportStandardScalerfromsklearn.ensembleimportRandomForestClassifier# Example usagesteps_classifier=[('scaler',StandardScaler()),('classifier',RandomForestClassifier(max_depth=5,n_estimators=95))]3.2 Deployment of the Scikit-learn Pipelinefromtdstone2.tdshypermodelimportHyperModelfromtdstone2.tdstoneimportTDStonesto=TDStone(schema_name=Param['database'],SEARCHUIFDBPATH=Param['user'])model_parameters={"target":'Y2',"column_categorical":['flag','Y2'],"column_names_X":['X1','X2','X3','X4','X5','X6','X7','X8','X9','flag']}model=HyperModel(tdstone=sto,metadata={'project':'test'},skl_pipeline_steps=steps_classifier,model_parameters=model_parameters,dataset=tdml.in_schema(Param['database'],'dataset_00'),id_row='ID',id_partition='Partition_ID',id_fold='FOLD',fold_training='train')# Model deployment outputs3.3 Local Execution for Validation/Debuggingexec(code_and_data['code'])local_model=MyModel(**code_and_data['arguments'])df_local=code_and_data['data']df_local['flag']=df_local['flag'].astype('category')df_local['Y2']=df_local['Y2'].astype('category')local_model.fit(code_and_data['data'])local_model.score(code_and_data['data'])Execution of the Deployed Hypermodel4.1 Models Trainingmodel.train()# Outputs: Trained models are inserted into the specified repositoryThis training operation launches as many training there are data partitions identified by the id_partition column list, and belonging to the training FOLD.4.2 Model Scoringmodel.score()# Outputs: Model scores are inserted into the specified scores tableThis runs the scoring on all the data, using the latest model available for the corresponding data partition.Reuse of the Hyper-segmented Model2. Retrieve the Hyper-segmented Modelsto=TDStone(schema_name=Param['database'],SEARCHUIFDBPATH=Param['user'])sto.list_hyper_models()# Outputs: List of hyper models with their metadataid='0286d259-ecde-4cd0-ae4a-bcb3191383d1'# Example model IDexisting_model=HyperModel(tdstone=sto)existing_model.download(id=id)Note that the model is not actually downloaded, but this just establish a connection between the model hosted in Vantage and the python client.3. Update the Trainingexisting_model.train()# Outputs: Updated trained models are inserted into the specified repository4. Update the Scoringexisting_model.score()# Outputs: Updated scores are inserted into the specified scores table