Package: cognite-replicator
Cognite Python ReplicatorCognite Replicator is a Python package for replicating data across Cognite Data Fusion (CDF) projects. This package is built on top of the Cognite Python SDK.
This component is Community content and not officially supported by Cognite. Bugs and changes will be fixed on a best effort basis. Feel free to open issues and pull requests, we will review them as soon as we can.Copyright 2023 Cognite ASPrerequisitesIn order to start using the Replicator, you need:Python3 (>= 3.6)Credentials for both the source and destination projects:CLIENT_ID ("Client ID from Azure")CLIENT_SECRET ("Client secret from Azure", only if using authentication via secret)CLUSTER ("Name of CDF cluster")TENANT_ID ("Tenant ID from Azure"PROJECT ("Name of CDF project")This is how you set the client secret as an environment variable on Mac OS and Linux:$exportSOURCE_CLIENT_SECRET=<yoursourceclientsecret>
$exportDEST_CLIENT_SECRET=<yourdestinationclientsecret>InstallationThe replicator is available onPyPI, and can also be executed .To run it from command line, run:pipinstallcognite-replicatorAlternatively, build and run it as a docker container. The image is avaible ondocker hub:dockerbuild-tcognite-replicator.Usage1. Run with a configuration file as a standalone scriptCreate a configuration file based on the config/default.yml and update the values corresponding to your environment
If no file is specified then replicator will use config/default.yml.via Pythonpython-mcognite.replicatorconfig/filepath.ymlor alternatively via docker
If no access to a browser, you should use the client secret authenticationdockerrun-eSOURCE_CLIENT_SECRET-eDEST_CLIENT_SECRET-v/absolute/path/to/config/config.yml:/config.ymlcognite-replicator/config.yml2. Setup as Python library2.1 Without configuration file and interactive loginIt will copy everything from source to destination and use your own credentials to run the code, you need to have the right permissions to read on the source project and write on the destination projectimportosimportyamlfromcognite.client.credentialsimportOAuthInteractivefromcognite.clientimportCogniteClient,ClientConfigfromcognite.replicatorimportassets,events,files,time_series,datapoints,sequences,sequence_rows# SOURCESOURCE_TENANT_ID="48d5043c-cf70-4c49-881c-c638f5796997"SOURCE_CLIENT_ID="1b90ede3-271e-401b-81a0-a4d52bea3273"SOURCE_PROJECT="publicdata"SOURCE_CLUSTER="api"# DESTINATIONDEST_TENANT_ID="d4febcbc-db24-4823-bffd-92fd05b9c6bc"DEST_CLIENT_ID="189e8b95-f1ce-47d2-aa66-4c2fe3567f91"DEST_PROJECT="sa-team"DEST_CLUSTER="bluefield"### Autogenerated variablesSOURCE_SCOPES=[f"https://{SOURCE_CLUSTER}.cognitedata.com/.default"]SOURCE_BASE_URL=f"https://{SOURCE_CLUSTER}.cognitedata.com"SOURCE_AUTHORITY_URL=f"https://login.microsoftonline.com/{SOURCE_TENANT_ID}"DEST_SCOPES=[f"https://{DEST_CLUSTER}.cognitedata.com/.default"]DEST_BASE_URL=f"https://{DEST_CLUSTER}.cognitedata.com"DEST_AUTHORITY_URL=f"https://login.microsoftonline.com/{DEST_TENANT_ID}"# ConfigBATCH_SIZE=10000# this is the max size of a batch to be postedNUM_THREADS=10# this is the max number of threads to be usedTIMEOUT=90PORT=53000SOURCE_CLIENT=CogniteClient(ClientConfig(credentials=OAuthInteractive(authority_url=SOURCE_AUTHORITY_URL,client_id=SOURCE_CLIENT_ID,scopes=SOURCE_SCOPES,),project=SOURCE_PROJECT,base_url=SOURCE_BASE_URL,client_name="cognite-replicator-source",))DEST_CLIENT=CogniteClient(ClientConfig(credentials=OAuthInteractive(authority_url=DEST_AUTHORITY_URL,client_id=DEST_CLIENT_ID,scopes=DEST_SCOPES,),project=DEST_PROJECT,base_url=DEST_BASE_URL,client_name="cognite-replicator-destination",))if__name__=="__main__":# this is necessary because threading#### Uncomment the resources you would like to copyassets.replicate(SOURCE_CLIENT,DEST_CLIENT)#events.replicate(SOURCE_CLIENT, DEST_CLIENT, BATCH_SIZE, NUM_THREADS)#files.replicate(SOURCE_CLIENT, DEST_CLIENT, BATCH_SIZE, NUM_THREADS)#time_series.replicate(SOURCE_CLIENT, DEST_CLIENT, BATCH_SIZE, NUM_THREADS)#datapoints.replicate(SOURCE_CLIENT, DEST_CLIENT)#sequences.replicate(SOURCE_CLIENT, DEST_CLIENT, BATCH_SIZE, NUM_THREADS)#sequence_rows.replicate(SOURCE_CLIENT, DEST_CLIENT, BATCH_SIZE, NUM_THREADS)2.2 Without configuration file and with client credentials authenticationIt will copy everything from source to destination and use your own credentials to run the code, you need to have the right permissions to read on the source project and write on the destination project
(in the example below, the secrets are stored as environment variables)importosfromcognite.client.credentialsimportOAuthClientCredentialsfromcognite.clientimportCogniteClient,ClientConfigfromcognite.replicatorimportassets,events,files,time_series,datapoints,sequences,sequence_rows# SOURCESOURCE_TENANT_ID="48d5043c-cf70-4c49-881c-c638f5796997"SOURCE_CLIENT_ID="1b90ede3-271e-401b-81a0-a4d52bea3273"SOURCE_CLIENT_SECRET=os.environ.get("SOURCE_CLIENT_SECRET")SOURCE_PROJECT="publicdata"SOURCE_CLUSTER="api"# DESTINATIONDEST_TENANT_ID="d4febcbc-db24-4823-bffd-92fd05b9c6bc"DEST_CLIENT_ID="189e8b95-f1ce-47d2-aa66-4c2fe3567f91"DEST_CLIENT_SECRET=os.environ.get("DEST_CLIENT_SECRET")DEST_PROJECT="sa-team"DEST_CLUSTER="bluefield"### Autogenerated variablesSOURCE_SCOPES=[f"https://{SOURCE_CLUSTER}.cognitedata.com/.default"]SOURCE_BASE_URL=f"https://{SOURCE_CLUSTER}.cognitedata.com"SOURCE_TOKEN_URL=f"https://login.microsoftonline.com/{SOURCE_TENANT_ID}/oauth2/v2.0/token"DEST_SCOPES=[f"https://{DEST_CLUSTER}.cognitedata.com/.default"]DEST_BASE_URL=f"https://{DEST_CLUSTER}.cognitedata.com"DEST_TOKEN_URL=f"https://login.microsoftonline.com/{DEST_TENANT_ID}/oauth2/v2.0/token"COGNITE_CONFIG_FILE="config/config.yml"# ConfigBATCH_SIZE=10000# this is the max size of a batch to be postedNUM_THREADS=10# this is the max number of threads to be usedTIMEOUT=90PORT=53000SOURCE_CLIENT=CogniteClient(ClientConfig(credentials=OAuthClientCredentials(token_url=SOURCE_TOKEN_URL,client_id=SOURCE_CLIENT_ID,scopes=SOURCE_SCOPES,client_secret=SOURCE_CLIENT_SECRET,),project=SOURCE_PROJECT,base_url=SOURCE_BASE_URL,client_name="cognite-replicator-source",))DEST_CLIENT=CogniteClient(ClientConfig(credentials=OAuthClientCredentials(token_url=DEST_TOKEN_URL,client_id=DEST_CLIENT_ID,scopes=DEST_SCOPES,client_secret=DEST_CLIENT_SECRET,),project=DEST_PROJECT,base_url=DEST_BASE_URL,client_name="cognite-replicator-destination",))if__name__=="__main__":# this is necessary because threading#### Uncomment the resources you would like to copyassets.replicate(SOURCE_CLIENT,DEST_CLIENT)#events.replicate(SOURCE_CLIENT, DEST_CLIENT, BATCH_SIZE, NUM_THREADS)#files.replicate(SOURCE_CLIENT, DEST_CLIENT, BATCH_SIZE, NUM_THREADS)#time_series.replicate(SOURCE_CLIENT, DEST_CLIENT, BATCH_SIZE, NUM_THREADS)#datapoints.replicate(SOURCE_CLIENT, DEST_CLIENT)#sequences.replicate(SOURCE_CLIENT, DEST_CLIENT, BATCH_SIZE, NUM_THREADS)#sequence_rows.replicate(SOURCE_CLIENT, DEST_CLIENT, BATCH_SIZE, NUM_THREADS)2.3 Alternative by having some elements of the configuration file as variableRefer todefault configuration fileorexample configuration filefor all keys in the configuration file
Start with client creation from either step 2.1 or 2.2if__name__=="__main__":# this is necessary because threadingconfig={"timeseries_external_ids":["pi:160670","pi:160623"],"datapoints_start":"100d-ago","datapoints_end":"now",}time_series.replicate(client_src=SOURCE_CLIENT,client_dst=DEST_CLIENT,batch_size=BATCH_SIZE,num_threads=NUM_THREADS,config=config,)datapoints.replicate(client_src=SOURCE_CLIENT,client_dst=DEST_CLIENT,external_ids=config.get("timeseries_external_ids"),start=config.get("datapoints_start"),end=config.get("datapoints_end"),)3. With configuration fileIt will use the configuration file to determine what will be copied
In this case, no need to create the client, it will be created based on what is in the configuration fileimportyamlfromcognite.replicator.__main__importmainimportosif__name__=="__main__":# this is necessary because threadingCOGNITE_CONFIG_FILE=yaml.safe_load("config/config.yml")os.environ["COGNITE_CONFIG_FILE"]=COGNITE_CONFIG_FILEmain()4. Local testingIt will use the configuration file to determine what will be copied
In this case, no need to create the client, it will be created based on what is in the configuration fileimportyamlimportsyssys.path.append("cognite-replicator")### Path of the local version of the replicator. Importing from outside of the current working directory requires sys.path, which is a list of all directories Python searches through.importosif__name__=="__main__":# this is necessary because threadingCOGNITE_CONFIG_FILE=yaml.safe_load("config/config.yml")os.environ["COGNITE_CONFIG_FILE"]=COGNITE_CONFIG_FILEmain()sys.path.remove("cognite-replicator")## Python will also search these paths for future projects unless they are removed. Removes unwanted search pathsDevelopmentChange the version in the files_version.pycd.ymlpyproject.tomlChangelogWondering about upcoming or previous changes? Take a look at theCHANGELOG.ContributingWant to contribute? Check outCONTRIBUTING.