Package: transpydata
TransPyDataA minimal framework for managing migrationsOverviewTransPyData implements a generic pipeline to perform migrations. It has 2 main components. First one isTransPyclass, which executes the migration pipeline according to a configuration. Second thedata servicesimplementations (IDataInput,IDataProcessandIDataOutput), this services manages how data is gathered, processed and sent to the new destination.TransPyTheTransPyclass manages the migration pipeline. It needs to be provided with an instance of:IDataInput: Manages the gathering of source data.IDataProcess: Manages data transformation and filtering prior to pass it to the data output.IDataOutput: Manages data sending to the new destination.NOTE: Data services overview belowApart from the data services there are other optional configurations:trans_py=TransPy()config={'datainput_source':[],# If working with single record pipeline this should be an iterable of data to feed IDataInput'datainput_by_one':False,# Enable single record pipeline on input'dataprocess_by_one':False,# Enable single record pipeline on processing'dataoutput_by_one':False,# Enable single record pipeline on output}trans_py.configure(config)The values in the snippet are the defaults, so by default the migration will move all migration data through the pipeline at once.All processing modeWhen all data services have the "by_one" flag toFalsethe migration will move all data at once through the pipeline. So theTransPyinstance will call the methodget_allofIDataInputconfigured to get all input data, with the response will callprocess_allofIDataProcess, and with the response ofIDataProcesswill callsend_allofIDataOutput. Finally a list withIDataOutputresults is returned byTransPy.Single record modeIf "by_one" flags areTruethe data are "queried" by one and moved through all the pipeline. TheIDataOutputreturn are accumulated and returned as list at the end of the processing, so theTransPyreturn type is the same.There are some additional cases, what ifdatainputanddataprocessare in "by_one" mode and dataoutput not? In this case the data is gathered and processed one by one, at the end of processing (IDataProcess) the results are accumulated and theIDataOutputis called with all data. Similar case whendataprocessanddataoutputare in "by_one" mode, data is gathered all at once and then piped one by one throughIDataProcessandIDataOutput.Data servicesunder constructionGetting startedTo start a migration create an instance ofTransPyand configure it. At least instances ofIDataInput,IDataProcessandIDataOutputneeds to be provided. Prior to starting the migration the data services might need to be configured too. Here is an code example:importjsonfromtranspydataimportTransPyfromtranspydata.config.datainput.MysqlDataInputimportMysqlDataInputfromtranspydata.config.dataprocess.NoneDataProcessimportNoneDataProcessfromtranspydata.config.dataoutput.RequestDataOutputimportRequestDataOutputdefmain():# Configure imputmysql_input=MysqlDataInput()config={'db_config':{'user':'root','password':'TryingTh1ngs','host':'localhost','port':'3306','database':'migration'},'get_one_query':None,# We'll go with all query'get_all_query':"""SELECT s.staff_Id, s.staff_name, s.staff_grade, m.module_Id, m.module_nameFROM staff sLEFT JOIN teaches t ON s.staff_Id = t.staff_IdLEFT JOIN module m ON t.module_Id = m.module_Id""",'all_query_params':{}# No where clause, no interpolation}mysql_input.configure(config)# Configure processnone_process=NoneDataProcess()# Configure outputrequest_output=RequestDataOutput()request_output.configure({'url':'http://localhost:8008','req_verb':'POST','headers':{'content-type':'application/json','accept-encoding':'application/json','x-app-id':'MT1'},'encode_json':True,'json_response':True})# Configure TransPytrans_py=TransPy()trans_py.datainput=mysql_inputtrans_py.dataprocess=none_processtrans_py.dataoutput=request_outputres=trans_py.run()print(json.dumps(res))if__name__=='__main__':main()Full working example could be found atexamples/mysql_to_http/, there is adocker-composeto launch an instance of mysql and a webserver.Custom data servicesFor now you can check the interfacesIDataInput,IDataProcessandIDataOutputto see what needs to be implemented in a custom data service.(I'll improve this section in the future)