Package: tapdata-cli
Tapdata Python SdkEnglishApplicable versiontapshell / Python-Sdk: ^2.3.0tapdata: ^2.9InstallInstall python3.7、pip；Runpip install tapdata_clito install sdk；if use Poetry, runpoetry add tapdata_clito install dependence.Initialserver="127.0.0.1:3000"access_code="3324cfdf-7d3e-4792-bd32-571638d4562f"fromtapdata_cliimportclicli.init(server,access_code)Multi-thread concurrency is not supportedIt will send a request to the server to obtain the identity information and save it as a global variable. Therefore, after multiple init the 'server' and 'access_code' variable will be overwritten.For situations where you need to use different servers and access_codes concurrently, use Python's multiprocess.DataSourceCreate DataSourceTo create DataSource by Python Sdk, you can do it by form or uri mode.Example for uri mode:fromtapdata_cliimportcliconnector="mongodb"# 数据源类型，mongodb mysql postgresmongo=cli.DataSource("mongodb",name="mongo")mongo.uri("mongodb://localhost:8080")# 数据源urimongo.save()or form mode:fromtapdata_cliimportclimongo=cli.DataSource("mongodb",name="mongo")mongo.host("localhost:27017").db("source").username("user").password("password").props("")mongo.type("source")# 数据源类型，source -> 只可作为源，target -> 只可作为目标，source_and_target -> 可以作为源和目标（默认）mongo.save()# success -> True, Failure -> FalseMore infomation to create DataSource, please readthis file.DataSource Listfromtapdata_cliimportclicli.DataSource().list()# return datastruct{"total":94,"items":[{"id":"","lastUpdBy":"","name":"","config":{},"connection_type":"","database_type":"","definitionScope":"","definitionVersion":"","definitionGroup":"","definitionPdkId":"",...}]}```### get datasource by id/name```pythonfromtapdata_cliimportclicli.DataSource(id="")# 根据id获取数据源信息cli.DataSource(name="")# 根据name获取数据源信息PipelineMigrate jobfromtapdata_cliimportcli# Create DataSourcecli.DataSource("mongodb",name="source").uri("").save()cli.DataSource("mongodb",name="target").uri("").save()# Create Source and target nodesource=cli.Source("source")target=cli.Sink("target")# copy all table by default;# copy by tables you want to, use table=[]# filter table, by table_resource=cli.Source("source",table=["table_1","table_2","table_3"],table_re="table_*")source.config({"migrateTableSelectType":"custom"})# change migrateTableSelectType: from all to custom# init pipeline installp=cli.Pipeline(name="example_job")p.readFrom(source).writeTo(target)# startp.start()# stopp.stop()# deletep.delete()# check statusp.status()# job listcli.Job.list()Job is th underlying implementation of Pipeline，so you can start job byjob.start()likepipeline.start()。# init job (get job info) by idfromtapdata_cliimportclijob=cli.Job(id="some id string")job.save()# success -> True, failure -> Falsejob.start()# success -> True, failure -> FalseSync jobBefore you start a sync job, update job mode tosync.fromtapdata_cliimportclicli.DataSource("mongodb",name="source").uri("").save()# create datasourcecli.DataSource("mongodb",name="target").uri("").save()# create targetp=cli.Pipeline(name="sync_job",mode="sync")# update to sync mode, or use p.dag.jobType = JobType.syncp.mode(cli.JobType.sync)# or you can update to sync mode by this way# read sourcep=p.readFrom("source.player")# source is db, player is tablep=p.readFrom(cli.Source("source",table="player",mode="sync"))# or you init a Source Node in sync mode# continue to complex operation next# filter cli.FilterType.keep (keep data) / cli.FilterType.delete (delete data)p=p.filter("id > 2",cli.FilterType.keep)# filerColumn cli.FilterType.keep (keep column) / cli.FilterType.delete (delete column)p=p.filterColumn(["name"],cli.FilterType.delete)# renamep=p.rename("name","player_name")# valueMapp=p.valueMap("position",1)# jsp=p.js("return record;")p.writeTo("target.player")# target is db, player is tablep.writeTo(cli.Sink("target",table="player",mode="sync")Master-slave Merge：p2=cli.Pipeline(name="source_2")# create pipeline which will be mergedp3=p.merge(p2,[('id','id')])# merge p2 and set joinkey, then writeTo a tablep3.writeTo("target.player")# target is db, player is tableInitial_syncIt's "initial_sync+cdc" mode by default. You can create a "initial_sync" job by this way:fromtapdata_cliimportclip=cli.Pipeline(name="")p.readFrom("source").writeTo("target")config={"type":"initial_sync"}# initial_sync jobp1=p.config(config=config)p1.start()Change config by config method like{"type": "cdc"}to create a initial_sync job。Python sdk has built-in param verification, you can update config by Pipeline.config, to see more configuration, you can seethis fileApi OperationCreate/Update Apiserverfromtapdata_cliimportcli# Createcli.ApiServer(name="test",uri="http://127.0.0.1:3000/").save()# Update# 1.Get ApiServer idapi_server_id=cli.ApiServer.list()["id"]# 2.Update ApiServer and savecli.ApiServer(id=api_server_id,name="test_2",uri="http://127.0.0.1:3000/").save()# delete apiservercli.ApiServer(id=api_server_id).delete()Publish Apifromtapdata_cliimportclicli.Api(name="test",table="source.player").publish()# source is db, player is tableUnpublish Apifromtapdata_cliimportclicli.Api(name="test").unpublish()Delete apifromtapdata_cliimportclicli.Api(name="test").delete()Check api statusfromtapdata_cliimportclicli.Api().status("test")# success -> "pending" or "active" / failure -> None