Package: sdi-utils
SAP Data Intelligence Utilites (sdi_utils) by thhapkesdi_utils PackageThis package includes a couple of helpers that ease the development of operators for use withSAP Data Intelligence (SAP DI).gensolutionfor generating solution packages from locally developed operators.textfield_parserthat parses textfields that could contain lists, maps, list of maps,  etc.set_loggingfor channeling logging output to a string for tap wiring it with separate monitortprogressfor a simple time keeping to check performance of certain operators (tasks)Installationpip install sdi_utilsSAP Data Intelligence Helper OperatorsI created a some operators which have been reused a couple of times. Maybe they provide a quick solution for your pipelines as well. Please be aware that many of them become obsolete with the next releases in particular when the loomingvtypesare introduced.You find the source code of all operators insrc/sdi_utils_operators/. They have been created outside a SAP Data Intelligence instance and withgensolutionthe solution has been created that can then directly been uploaded. But all generated solutions you also find in the foldersdi_utils/solution/operators/.All operators require the modules packaged insdi_utils, so you have to add at least this to your docker image. For additional packages have a look at the tags added to each operator.The following list might not encompass all operators you find in the source or solutions folder. Because I am not been able to maintain this document in sync with the operators I create constantly.OperatorDescriptioncsv_dfConverts a csv byte/string stream into a pandas DataFramecsv_dictConverts a csv byte/string stream into a list of dictionaries with header as keycsv_tableConverts a csv byte/string stream into a 2 dimensional array. Used for the beta Hana Operator only.df_csvConverts a DataFrame into a csv-string for saving with WriteFile-operatordf_tableConverts a DataFrame into a 2 dimensional array with the according attributes used for writing in a HANA Database (Beta-operatordict_dfConverts a dictionary into DataFramedict_jsonConverts a dictionary into a JSON-stringdict_tableConverts a dictionary into a 2-dimensional arrayfilter_dateWhen the date is part of the filename (format: yyyy-mm-dd) then files can be selected with dates within the period stated in the configurationcount_gateDecision gate that triggers next processing step only after the number stated in the configuration has been reached. The number can also be set dynamically either directly via the limit port or via attribute port. This operator is mostly used to terminate a pipeline.HTTPdownloadDownloads the file given in the urljson_dfConverts a json-string into a DataFramejson_dfConverts a json-string into a dictionaryline_arrayConverts a byte-stream into an 1-dimensional arraytable_csvConverts a 2-dimensional array into a csv. Using the attributes of thebeta-type table for the header.Operator Testing and ModifyingIf you like to test and modify the operators outside of Data Intelligence and create the operator-solution automatically you have to install vctl and sdi_utils. Then you can add the following code-snippet at the end of the operator-code:if __name__ == '__main__':
    test_operator()
    subprocess.run(["rm", '-r','<projectfolder>/solution/operators/sdi_utils_operators' + api.config.version])
    gs.gensolution(os.path.realpath(__file__), api.config, inports, outports)
    solution_name = api.config.operator_name + '_' + api.config.version
    subprocess.run(["vctl", "solution", "bundle", '<projectfolder>/solution/operators/sdi_utils_operators_' + api.config.version, "-t", solution_name])
    subprocess.run(["mv", solution_name + '.zip', '../../../solution operators'])Githubgithub repository