Package: pycarol
NoteFor the latest source, discussion, etc, please visit theGitHub repositoryPyCarolContentsPyCarolGetting StartedInitializing pyCarolUsing API KeyGood practice using tokenIngesting dataFrom both Staging Tables and Data Models (CDS Layer)From both Staging Tables and Data Models (BQ Layer)From Data Models (RT Layer): Filter queriesFrom Data Models (RT Layer): Named queriesSending dataLoggingSettingsUseful FunctionsRelease processGetting StartedRunpip install pycarolto install the latest stable version fromPyPI.Documentation for the latest releaseis hosted on readthedocs.This will install the minimal dependencies. To install pyCarol with thedataframesdependencies usepip install pycarol[dataframe], or to install with dask+pipeline dependencies usepip install pycarol[pipeline,dask]The options we have are:complete,dataframe,onlineapp,dask,pipelineTo install from source:pip install-rrequirements.txtto install the minimal requirements;pip install-e.".[dev]"to install the minimal requirements + dev libs;pip install-e.".[pipeline]"to install the minimal requirements + pipelines dependencies;pip install-e.".[complete]"to install all dependencies;Initializing pyCarolCarol is the main object to access pyCarol and all Carol’s APIs.frompycarolimportPwdAuth,Carolcarol=Carol(domain=TENANT_NAME,app_name=APP_NAME,auth=PwdAuth(USERNAME,PASSWORD),organization=ORGANIZATION)wheredomainis the tenant name,app_nameis the Carol’s app name, if any,authis the authentication method to be used (using user/password in this case) andorganizationis the organization
one wants to connect. Carols’s URL is build aswww.ORGANIZATION.carol.ai/TENANT_NAMEIt is also possible to initialize the object with a token generated via user/password. This is useful when creating an
online app that interacts with CarolfrompycarolimportPwdKeyAuth,Carolcarol=Carol(domain=TENANT_NAME,app_name=APP_NAME,auth=PwdKeyAuth(pwd_auth_token),organization=ORGANIZATION)Using API KeyTo use API keys instead of username and password:frompycarolimportApiKeyAuth,Carolcarol=Carol(domain=DOMAIN,app_name=APP_NAME,auth=ApiKeyAuth(api_key=X_AUTH_KEY),connector_id=CONNECTOR,organization=ORGANIZATION)In this case one changes the authentication method toApiKeyAuth. Noticed that one needs to pass theconnector_idtoo. An API key is always associated to a connector ID.It is possible to use pyCarol to generate an API keyfrompycarolimportPwdAuth,ApiKeyAuth,Carolcarol=Carol(domain=TENANT_NAME,app_name=APP_NAME,organization=ORGANIZATION,auth=PwdAuth(USERNAME,PASSWORD),connector_id=CONNECTOR)api_key=carol.issue_api_key()print(f"This is a API key{api_key['X-Auth-Key']}")print(f"This is the connector Id{api_key['X-Auth-ConnectorId']}")To get the details of the API key you can do:details=carol.api_key_details(APIKEY,CONNECTORID)Finally, to revoke an API key:carol.api_key_revoke(CONNECTORID)Good practice using tokenNever write in plain text your password/API token in your application. Use environment variables. pyCarol can use
environment variables automatically. When none parameter is passed to the Carol constructor pycarol will look for:CAROLTENANTfor domainCAROLAPPNAMEfor app_nameCAROL_DOMAINfor environmentCAROLORGANIZATIONfor organizationCAROLAPPOAUTHfor authCAROLCONNECTORIDfor connector_idCAROLUSERfor carol user emailCAROLPWDfor user password.e.g., one can create a.envfile like this:CAROLAPPNAME=myAppCAROLTENANT=myTenantCAROLORGANIZATION=myOrganizationCAROLAPPOAUTH=myAPIKeyCAROLCONNECTORID=myConnectorand thenfrompycarolimportCarolfromdotenvimportload_dotenvload_dotenv(".env")#this will import these env variables to your execution.carol=Carol()Ingesting dataFrom both Staging Tables and Data Models (CDS Layer)Use this method when you need to read most of the records and columns from the source.frompycarolimportCarol,Stagingstaging=Staging(Carol())df=staging.fetch_parquet(staging_name="execution_history",connector_name="model")From both Staging Tables and Data Models (BQ Layer)Use this method when you need to read only a subset of records and columns or when
data transformation is needed.frompycarolimportBQ,Carolbq=BQ(Carol())query_str="SELECT * FROM stg_connectorname_table_name"results=bq.query(query_str)In case one needs a service account with access to BigQuery, the following code can be
used:frompycarolimportCarolfrompycarol.bigqueryimportTokenManagertm=TokenManager(Carol())service_account=tm.get_token().service_accountAfter each execution ofBQ.query, theBQobject will have an attribute calledjob. This attribute is of typebigquery.job.query.QueryJoband may be useful for
monitoring/debug jobs.PyCarol provides access to BigQuery Storage API also. It allows for much faster reading
times, but with limited querying capabilities. For instance, only tables are readable,
so ‘ingestion_stg_model_deep_audit’ is ok, but ‘stg_model_deep_audit’ is not (it is a
view).frompycarolimportBQStorage,Carolbq=BQStorage(Carol())table_name="ingestion_stg_model_deep_audit"col_names=["request_id","version"]restriction="branch = '01'"sample_size=1000df=bq.query(table_name,col_names,row_restriction=restriction,sample_percentage=sample_size,return_dataframe=True)From Data Models (RT Layer): Filter queriesUse this when you need low latency (only if RT layer is enabled).frompycarol.filterimportTYPE_FILTER,TERM_FILTER,FilterfrompycarolimportQueryjson_query=Filter.Builder()\.must(TYPE_FILTER(value='ratings'+"Golden"))\.must(TERM_FILTER(key='mdmGoldenFieldAndValues.userid.raw',value='123'))\.build().to_json()FIELDS_ITEMS=['mdmGoldenFieldAndValues.mdmaddress.coordinates']query=Query(carol,page_size=10,print_status=True,only_hits=True,fields=FIELDS_ITEMS,max_hits=200).query(json_query).go()query.resultsThe result will be200hits of the queryjson_queryabove, the pagination will be 10, that means in each response
there will be 10 records. The query will return only the fields set inFIELDS_ITEMS.The parameteronly_hits = Truewill make sure that only records into the path$hits.mdmGoldenFieldAndValueswill return.
If one wants all the response useonly_hits = False. Also, if your filter has an aggregation, one should useonly_hits = Falseandget_aggs=True, e.g.,frompycarolimportQueryfrompycarol.filterimportTYPE_FILTER,Filter,CARDINALITYjson_query=Filter.Builder()\.must(TYPE_FILTER(value='datamodelname'+"Golden"))\.aggregation(CARDINALITY(name='cardinality',params=["mdmGoldenFieldAndValues.taxid.raw"],size=40))\.build().to_json()query=Query(carol,get_aggs=True,only_hits=False)query.query(json_query).go()query.resultsFrom Data Models (RT Layer): Named queriesfrompycarolimportQuerynamed_query='revenueHist'# named query nameparams={"bin":"1d","cnpj":"24386434000130"}#query parameters to send.results=Query(carol).named(named_query,params=params).go().resultsIt is possible to use all the parameters used in the filter query, i.e.,only_hits,save_results, etc. For more information for the possible input parameters check the docstring.What if one does not remember the parameters for a given named query?named_query='revenueHist'# named query nameQuery(carol).named_query_params(named_query)>{'revenueHist':['*cnpj','dateFrom','dateTo','*bin']}#Parameters starting by * are mandatory.Sending dataThe first step to send data to Carol is to create a connector.frompycarolimportConnectorsconnector_id=Connectors(carol).create(name='my_connector',label="connector_label",group_name="GroupName")print(f"This is the connector id:{connector_id}")With the connector Id on hands we can create the staging schema and then create the staging table. Assuming we have
a sample of the data we want to send.frompycarolimportStagingjson_ex={"name":'Rafael',"email":{"type":"email","email":'rafael@totvs.com.br'}}staging=Staging(carol)staging.create_schema(staging_name='my_stag',data=json_ex,crosswalk_name='my_crosswalk',crosswalk_list=['name'],connector_name='my_connector')The json schema will be in the variableschema.schema. The code above will create the following schema:{'mdmCrosswalkTemplate':{'mdmCrossreference':{'my_crosswalk':['name']}},'mdmFlexible':'false','mdmStagingMapping':{'properties':{'email':{'properties':{'email':{'type':'string'},'type':{'type':'string'}},'type':'nested'},'name':{'type':'string'}}},'mdmStagingType':'my_stag'}To send the data  (assuming we have a json with the data we want to send).frompycarolimportStagingjson_ex=[{"name":'Rafael',"email":{"type":"email","email":'rafael@totvs.com.br'}},{"name":'Leandro',"email":{"type":"email","email":'Leandro@totvs.com.br'}},{"name":'Joao',"email":{"type":"email","email":'joao@rolima.com.br'}},{"name":'Marcelo',"email":{"type":"email","email":'marcelo@totvs.com.br'}}]staging=Staging(carol)staging.send_data(staging_name='my_stag',data=json_ex,step_size=2,connector_id=connectorId,print_stats=True)The parameterstep_sizesays how many registers will be sent each time. Remember the the max size per payload is
5MB. The parameterdatacan be a pandas DataFrame.OBS: It is not possible to create a mapping using pycarol. The Mapping has to be done via the UILoggingTo log messages to Carol:frompycarolimportCarol,CarolHandlerimportlogginglogger=logging.getLogger(__name__)logger.setLevel(logging.DEBUG)carol=CarolHandler(Carol())carol.setLevel(logging.INFO)logger.addHandler(carol)logger.debug('This is a debug message')#This will not be logged in Carol. Level is set to INFOlogger.info('This is an info message')logger.warning('This is a warning message')logger.error('This is an error message')logger.critical('This is a critical message')These methods will use the current long task id provided by Carol when running your application.
For local environments you need to set that manually first on the beginning of your code:importosos.environ['LONGTASKID']=task_idWe recommend to log only INFO+ information in Carol. If no TASK ID is passed it works as a Console Handler.SettingsWe can use pyCarol to access the settings of your Carol App.frompycarol.appsimportAppsapp=Apps(carol)settings=app.get_settings(app_name='my_app')print(settings)The settings will be returned as a dictionary where the keys are the parameter names and the values are
the value for that parameter. Please note that your app must be created in Carol.Useful Functionstrack_tasks: Track a list of tasks.frompycarolimportCarolfrompycarol.functionsimporttrack_taskscarol=Carol()defcallback(task_list):print(task_list)track_tasks(carol=carol,task_list=['task_id_1','task_id_2'],callback=callback)Release processOpen a PR with your change formasterbranch;Once approved, merge intomaster;In case there are any changes to the default release notes, please update them