Package: ditat-etl
Ditat ETLMultiple tools and utilities for ETL pipelines and others.DatabasesUseful wrappers for databases and methods to execute queries.PostgresIt is compatible with pandas.DataFrame interaction, either reading as dataframes and pushing to the db.fromditat_etl.databasesimportPostgresconfig={"database":"xxxx","user":"xxxx","password":"xxxx","host":"xxxxx","port":"xxxx"}p=Postgres(config)The main base function is query.p.query(query_statement:listorstr,df:bool=False,as_dict:bool=False,commit:bool=True,returning:bool=True,mogrify:bool=False,mogrify_tuple:tupleorlist=None,verbose=False)This function is a workaround of pandas.to_sql() which drops the table before inserting.
It really works like an upsert and it gives you the option to do nothing or update on the column(s) constraint.p.insert_df_to_sql(df:pd.DataFrame,tablename:str,commit=True,conflict_on:strorlist=None,do_update_columns:boolorlist=False,verbose=False):This one is similar, it lets you "upsert" without necessarily having a primary key or constraint.
Ideally use the previous method.p.update_df_to_sql(df:pd.DataFrame,tablename:str,on_columns:strorlist,insert_new=True,commit=True,verbose=False,overwrite=False):This class also has some smaller utitlies for describing tables, getting information,
creating tables and indexes among others.SalesforceSalesforceObjSalesforce wrapper adding functionalities to Simple Salesforce.You can connect with login crendentials or through the Oauth2 protocol with
a refresh token and client crendentials.fromditat_etl.salesforceimportSalesforceObjsf_config={"organizationId":"xxx","password":"xxx","security_token":"xxx","username":"xxx",# Session Id for refresh token"session_id":"xxx"}# With login credentialssf=Salesforce(sf_config)# With Oauth2 refresh token and client credentialssf=SalesforceObj(sf_config,client_id='xxx',client_secret='xxx',)The core of this class is the query method.sf.query(tablename='xxx',columns=None,limit=None,df=False,date_from=None,date_to=None,date_window=None,date_window_variable='LastModifiedDate',verbose=False,timeout=None,max_columns=100,)Most times you can directly use the parallel execution.sf.query_parallel(tablename='xxx',columns=None,limit=None,df=False,date_window=None,date_window_variable='LastModifiedDate',verbose=False,n_chunks=4)Update recordssf.update(tablename="xxx",record_list=[{'Id':'11111','field1':'1111'}],batch_size=1000)Insert records (update as an option)sf.insert_df(tablename='xxx',dataframe=pd.DataFrame,batch_size=10000,update=True,conflict_on='Name',return_response=False,)Entity Resolution and DeduplicationThe Matcher allows you to match two datasets on certain criteria
and also find duplicates.Matcherfromditat_etl.utils.entity_resolutionimportMatchermatcher=Matcher(exact_domain=False)Entity ResolutionSet both datasets as dataframes. Specify the names of the columns to use
as criteria for matching# Set df 1matcher.set_df(data=pd.DataFrame,name='df_1',index='id',domain=None,address=None,phone=None,country=None,entity_name=None)# Set df 2matcher.set_df(data=pd.DataFrame,name='df_2',index='id',domain=None,address=None,phone=None,country=None,entity_name=None)# Runmatches=matches.run(save=False,verbose=True,deduping=False,match_type_included=None,match_type_excluded=None,match_count_th=3)Deduplicationresults=matcher.dedupe(data=pd.DataFrame,index='id',domain=None,address=None,phone=None,country=None,entity_name=None,save=True,match_type_included=None,match_type_excluded=None,match_count_th=3,include_self=True)TimeItThe TimeIt class helps you time functions, methods, blocks of code.fromditat_etl.timeimportTimeItA)@TimeItdeff():passf()>>ftakes:0.0006sec.B)@TimeIt(decimals=8)deff():passf()>>ftakes:0.00000003sec.C)withTimeIt():time.sleep(0.3)>>indentedblocktakes:0.3005sec.D)classFoo:@TimeIt()# It has to be a callable for classesdefm(self):passFoo().m()>>mtakes:0.0sec.EnrichmentPeopleDataLabsfromditat_etl.utils.enrichmentimportPeopleDataLabspdl=PeopleDataLabs(api_key='xxx',check_existing_method='s3',bucket_name='xxx',)You can:Enrich companySearch companyEnrich personSearch personr=pdl.enrich_company(min_likelihood=5,required='website',save=True,check_existing=True,s3_recalculate=False,company_name='companyx')'''PLEASE CHECK https://docs.peopledatalabs.com/docsFOR MORE DETAILS'''Urlfunctionsfromditat_etl.url.functionsimporteval_url,extract_domaindata=['laskdj','www.google.com','accounts.google.com','https://www.emol.com/economia/']fordindata:r=extract_domain(d)print(r)print('####')r=eval_url(data)print(r)None
google.com
accounts.google.com
emol.com####Initializing4workers.
eval_urltakes:1.3309sec.{'laskdj':None,'www.google.com':'google.com','accounts.google.com':'accounts.google.com','https://www.emol.com/economia/':'emol.com'}UrlExtension of module requests/urllib3 for Proxy usage and Bulk usage.High-level usagefromditat_etlimporturlresponse=url.get('https://google.com')# You can pass the same parameters as the library requests and other special parameters.# Check low level usage for more details.Low-level usagefromditat_etl.urlimportUrlu=Url()We use the logging module and it is set by default with 'DEBUG'.
You can change this parameter to any allowed levelu=Url(debug_level='WARNING')# Just an exampleManage your proxiesu.add_proxies(n=3)# Added 3 new proxies (not necessarily valid) to self.proxiesu.clean_proxies()# Multithreaded to validate and keep only valid proxies.print(u.proxies)# You can also u.proxies = [], set them manually but this is not recommended.Main functionalitydefrequest(queue:strorlist,expected_status_code:int=200,n_times:int=1,max_retries:int=None,use_proxy=False,_raise=True,***kwargs):Examplesresult=u.request('https://google.com')result=u.request(queue=['https://google.com','htttps://facebook.com'],use_proxy=True)# You can also pass optional parameter valid por a requests "Request"importjsonresult=u.request(queue='https://example.com',method='post',data=json.dumps({'hello':'world'}))