Package: dativatools
Dativa ToolsProvides useful libraries for processing large data sets.
Developed by the team atwww.dativa.comas we find them useful in our projects.The key libraries included here are:dativa.tools.aws.S3Csv2Parquet - an AWS Glue based tool to transform CSV files to Parquet filesdativa.tools.aws.AthenaClient - provide a simple wrapper to execute Athena queries and create tables. When combined
with the S3Csv2Parquet handler can automatically change Athena outputs to Parquet formatdativa.tools.aws.PipelineClient - client to interact with the Pipeline API. When provided an api key, source S3
location, destination s3 location, and rules, it will clean the source file and post it to destination.dativa.tools.aws.S3Client - a wrapper for AWS's boto library for S3 enabling easier iteration over S3 files and
multiple deletions, as well as uploading multiple filesdativa.tools.SQLClient - a wrapper for any PEP249 compliant database client with logging and splitting of queriesdativa.tools.pandas.CSVHandler - improved CSV handling for Pandasdativa.tools.pandas.ParquetHandler - improved Parquet handling for pandasdativa.tools.pandas.Shapley - Shapley attribution modelling using pandas DataFramesThere are also some useful support functions for Pandas date and time handling.
As well as a function to save data out in a format suitable for Athena.Installationpip install dativatoolsNote that Dativa Tools uses loose binding to required libraries and the required binaries are thus not automatically installed with the package and you can import classes and functions from dativa tools without the required libraries installed. An ImportError is only raised at runtime if you are looking to use a function that depends on another package that is not installed.The required libraries are listed in requirements.txt and include:pyarror for ParquetHandlerpandas for all of the pandas extensionsawsretry and boto3 for any functions requiring AWS accesss3fs for many functions using S3blist for Shapleypycryptodome for encrypting CSVs in CSVHandlerchardet for sniffing encodings in CSVHandlerrequests for PipelineClientDescriptiondativa.tools.aws.AthenaClientAn easy to use client for AWS Athena that will create tables from S3 buckets (using AWS Glue) and run queries against these tables. It support full customisation of SerDe and column names on table creation.Examples:Creating tablesThe library creates a temporary Glue crawler which is deleted after use, and will also create the database if it does not exist.fromdativa.tools.awsimportAthenaClientac=AthenaClient("us-east-1","my_athena_db")ac.create_table(table_name='my_first_table',crawler_target={'S3Targets':[{'Path':'s3://my-bucket/table-data'}]})# Create a table with a custom SerDe and column names, typical for CSV filesac.create_table(table_name='comcast_visio_match',crawler_target={'S3Targets':[{'Path':'s3://my-bucket/table-data-2','Exclusions':['**._manifest']}]},serde='org.apache.hadoop.hive.serde2.OpenCSVSerde',columns=[{'Name':'id','Type':'string'},{'Name':'device_id','Type':'string'},{'Name':'subscriber_id','Type':'string'}])Running queriesfromdativa.tools.awsimportAthenaClientac=AthenaClient("us-east-1","my_athena_db")ac.add_query(sql="select * from table",name="My first query",output_location="s3://my-bucket/query-location/")ac.wait_for_completion()Fetch results of queryfromdativa.tools.awsimportAthenaClientac=AthenaClient("us-east-1","my_athena_db")query=ac.add_query(sql="select * from table",name="My first query",output_location="s3://my-bucket/query-location/")ac.wait_for_completion()ac.get_query_result(query)Running queries with the output in Parquet and create an Athena tablefromdativa.tools.awsimportAthenaClient,S3Csv2Parquetscp=S3Csv2Parquet(region="us-east-1",template_location="s3://my-bucket/glue-template-path/")ac=AthenaClient("us-east-1","my_athena_db",s3_parquet=scp)ac.add_query(sql="select * from table",name="my query that outputs Parquet",output_location="s3://my-bucket/query-location/",parquet=True)ac.wait_for_completion()ac.create_table({'S3Targets':[{'Path':"s3://my-bucket/query-location/"}]},table_name="query_location")dativa.tools.aws.S3ClientAn easy to use client for AWS S3 that adds some functionality
Examples:S3LocationClass that parses out an S3 location from a passed string. Subclass ofstrso supports most string operations.Also contains properties .bucket, .key, .path, .prefix and method .join()param s3_str: string representation of s3 location, accepts most common formatseg:
    - 's3://bucket/folder/file.txt'
    - 'bucket/folder'
    - 'http[s]://s3*.amazonaws.com/bucket-name/'
also accepts None if using `bucket` and `key` keywordparam bucket: ignored if s3_str is not None. can specify only bucket for
bucket='mybucket' - 's3://mybucket/' or in conjuction withkeyparam key: ignored if s3_str is not None. Bucket must be set.
bucket='mybucket', key='path/to/file' - 's3://mybucket/path/to/file'param ignore_double_slash: default False. If true allows s3 locations containing '//'
these are valid s3 paths, but typically result from mistaken joinsBatch deleting of files on S3fromdativa.tools.awsimportS3Client# Delete all files in a folders3=S3Client()s3.delete_files(bucket="bucket_name",prefix="/delete-this-folder/")# Delete only .csv.metadata files in a folders3=S3Client()s3.delete_files(bucket="bucket_name",prefix="/delete-this-folder/",suffix=".csv.metadata")Copy files from folder in local filesystem to s3 bucketfromdativa.tools.awsimportS3Clients3=S3Client()s3.put_folder(source="/home/user/my_folder",bucket="bucket_name",destination="backup/files")# Copy all csv files from folder to s3s3.put_folder(source="/home/user/my_folder",bucket="bucket_name",destination="backup/files",file_format="*.csv")dativa.tools.SQLClientA SQL client that wraps any PEP249 compliant connection object and provides detailed logging and simple query execution.It takes the following parameters when instantaited:db_connection - a PEP257 compatible database connectionlogger - the logger to uselogging_level - the level at which to log most outputlog_query_text - whether to log all of the query texthumour - if set to true output jokes to pass the time of day waiting for queries to completeexecute_queryRuns a query and ignores any outputParameters:query - the query to run, either a SQL file or a SQL queryparameters - a dict of parameters to substitute in the queryreplace - a dict or items to be replaced in the SQL textfirst_to_run - the index of the first query in a mult-command query to be executedexecute_query_to_dfRuns a query and returns the output of the final statement in a DataFrame.Parameters:query - the query to run, either a SQL file or a SQL queryparameters - a dict of parameters to substitute in the queryreplace - a dict or items to be replaced in the SQL textfirst_to_run - the index of the first query in a mult-command query to be executeddef execute_query_to_csvRuns a query and writes the output of the final statement to a CSV file.Parameters:query - the query to run, either a SQL file or a SQL querycsvfile - the file name to save the query results toparameters - a dict of parameters to substitute in the queryreplace - a dict or items to be replaced in the SQL textExample codeimportosimportpsycopg2fromdativa.toolsimportSqlClient# set up the SQL client from environment variablessql=SqlClient(psycopg2.connect(database=os.environ["DB_NAME"],user=os.environ["USER"],password=os.environ["PASSWORD"],host=os.environ["HOST"],port=os.environ["PORT"],client_encoding="UTF-8",connect_timeout=10))# create the full schedule tabledf=sql.execute_query_to_df(query="sql/my_query.sql",parameters={"start_date":"2018-01-01","end_date":"2018-02-01"})dativa.tools.log_to_stdoutA convenience function to redirect a specific logger and its children to stdoutimportloggingfromdativa.toolsimportlog_to_stdoutlog_to_stdout("dativa.tools",logging.DEBUG)dativa.tools.pandas.CSVHandlerA wrapper for pandas CSV handling to read and write dataframes
that is provided in pandas with consistent CSV parameters and
sniffing the CSV parameters automatically.
Includes reading a CSV into a dataframe, and writing it out to a string.Parametersbase_path: the base path for any CSV file read, if passed as a stringdetect_parameters: whether the encoding of the CSV file should be automatically detectedencoding: the encoding of the CSV files, defaults to UTF-8delimiter: the delimeter used in the CSV, defaults to ,header: the index of the header row, or -1 if there is no headerskiprows: the number of rows at the beginning of file to skipquotechar: the quoting character to use, defaults to ""include_index: specifies whether the index should be written out, default to Falsecompression: specifies whether the data should be compressed, default to 'infer', current support for writing out gzip and zip compressed filesnan_values: an array of possible NaN values, the first of which is used when writign out, defaults to Noneline_terminator: the line terminator to be usedquoting: the level of quoting, defaults to QUOTE_MINIMALdecimal: the decimal character, defaults to '.'chunksize: if specified the CSV is written out in chunksaes_key: bytes, allowable lengths are 16, 24, 32zipfile_compression: the type of zip compressions to use, default to ZIP_DEFLATEDfor decrypting and encrypting CSVs when passing to a dataframe, this uses AES CFB encryption via Pycryptodomeaes_iv: bytes, must have length of 16the initialization vector for the AES CFB encryption via Pycryptodome. If aes_key is specified and this is
not, it will auto-generate an iv and prefix it to the encrypted bytes.load_dfOpens a CSV file using the specified configuration for the class and raises an exception if the encoding is unparseable.
Detects if base_path is an S3 location and loads data from there if required.Parameters:file - File path. Should begin with 's3://' to load from S3 location.force_dtype - Force data type for data or columns, defaults to Nonekwargs - any of the keyword arguments used to create the class can also be passed to load_dfReturns:dataframesave_dfWrites a formatted string from a dataframe using the specified configuration for the class the file. Detects if base_path is an S3 location and saves data there if required.Parameters:df - Dataframe to savefile - File path. Should begin with 's3://' to save to an S3 location.kwargs - any of the keyword arguments used to create the class can also be passed to save_dfdf_to_stringReturns a formatted string from a dataframe using the specified configuration for the class.Parameters:df - Dataframe to convert to stringkwargs - any of the keyword arguments used to create the class can also be passed to df_to_stringReturns:stringExample codefromdativa.tools.pandasimportCSVHandler# Create the CSV handlercsv=CSVHandler(base_path='s3://my-bucket-name/')# Load a filedf=csv.load_df('my-file-name.csv')# Create a stringstr_df=csv.df_to_string(df)# Save a filecsv.save_df(df,'another-path/another-file-name.csv')Support functions for Pandasdativa.tools.pandas.is_numeric - a function to check whether a series or string is numericdativa.tools.pandas.string_to_datetime - a function to convert a string, or series of strings to a datetime, with a strptime date format that supports nanosecondsdativa.tools.pandas.datetime_to_string - a function to convert a datetime, or a series of datetimes to a string, with a strptime date format that supports nanosecondsdativa.tools.pandas.format_string_is_valid - a function to confirm whether a strptime format string returns a datedativa.tools.pandas.get_column_name - a function to return the name of a column from a passed column name or index.dativa.tools.pandas.get_unique_column_name - a function to return a unique column name when adding new columns to a DataFramedativa.tools.pandas.ParquetHandlerParquetHandler class, specify path of parquet file,
and get pandas dataframe for analysis and modification.param base_path                       : The base location where the parquet_files are stored.type base_path                        : strparam row_group_size                  : The size of the row groups while writing out the parquet file.type row_group_size                   : intparam use_dictionary                  : Specify whether to use boolean encoding or nottype use_dictionary                   : boolparam use_deprecated_int96_timestamps : Write nanosecond resolution timestamps to INT96 Parquet format.type use_deprecated_int96_timestamps  : boolparam coerce_timestamps               : Cast timestamps a particular resolution. Valid values: {None, 'ms', 'us'}type coerce_timestamps                : strparam compression                     : Specify the compression codec.type compression                      : strfromdativa.tools.pandasimportCSVHandler,ParquetHandler# Read a parquet filepq_obj=ParquetHandler()df_parquet=pq_obj.load_df('data.parquet')# save a csv_file to parquetcsv=CSVHandler(csv_delimiter=",")df=csv.load_df('emails.csv')pq_obj=ParquetHandler()pq_obj.save_df(df,'emails.parquet')save_dfSaves the df as parquet to the file path given to it, similar to CSVHandler save_df.Parametersparam df                              : A pandas dataframe to write to original file location of parquet file.type df                               : pandas.DataFrameparam row_group_size                  : The size of the row groups while writing out the parquet file.type row_group_size                   : intparam use_deprecated_int96_timestamps : Write nanosecond resolution timestamps to INT96 Parquet format.type use_deprecated_int96_timestamps  : boolparam coerce_timestamps               : Cast timestamps a particular resolution. Valid values: {None, 'ms', 'us'}type coerce_timestamps                : strparam compression                     : Specify the compression codec.type compression                      : strparam schema                          : Used to set the desired schema for pyarrow table, if not provided schema is inferredtype schema                           : pyarrow.lib.Schema or dictparam infer_other_dtypes              : Used when schema is specified. When True, if there are columns not specified in schema then their dtypes are inferred. When false, if there are columns not specified in schema then raise an error. Default behaviour is False.type infer_other_dtypes               : boolFor convenience, ParquetHandler allows a python dict to be passed to the schema argument. The dict should have column names as the keys and desired types as the values. A dict or schema for only some of the columns may be passed, the types for the rest of the columns will then be inferred. The types are inferred by looking at the types for the non-null values in each column. An error is raised if there multiple types in each column.Example code of how to pass a dict to the schema argument. In this example, only columnscol1andcol2are given types, any other columns will have their types inferred.pq_obj=ParquetHandler()dict_schema={'col1':str,'col3':int}pq_obj.save_df(test_df,new_file_path,schema=dict_schema)Example code on how to generate pyarrow.lib.schema objects and how to pass the schema to save_df.pq_obj=ParquetHandler()fields=[pa.field("col1",pa.int64()),pa.field("col2",pa.string())]my_schema=pa.schema(fields)pq_obj.save_df(test_df,new_file_path,schema=my_schema)dativa.tools.pandas athena_partitionA function to handle partitioning and saving a pandas DataFrame in a format compatible with athena. Using one or more specified column from the DataFrame being saved.Parametersparam df                      : The data frame to be partitionedtype df                       : pandas.DataFrameparam partition_categories    : The columns to partition the data ontype partition_categories     : listparam file_handler            : The appropriate file handler to save the data, currently tested for dativa CSVHandler and ParquetHandler support, other handlers are untestedtype file_handler             : objparam suffix                  : The extension the file should be saved with, .csv for csv, and .parquet for parquettype suffix                   : strparam columns_to_keep         : Columns to keep from the data frame, if not supplied default behaviour is to keep all columnstype columns_to_keep          : listparam date_time_format        : To minimise chances of overwrite the saved files contain the date time of when this function was called, this param specifies the format of the date time in strftime formattype date_time_format         : strparam name                    : If provided all files filename will start with thistype name                     : strparam partition_string        : Allows formatting folder names, will be dependant on how many partition categories there are, defaults to creating folders and sub folders in order of partitioningtype partition_string         : strparam partition_dtypes        : Can pass argument to set the dtype of a particular column, to ensure proper grouping, also doubles to checking the column doesnt contain values of an unexpected dtypetype partition_dtypes         : listparam kwargs                  : Any additional key word arguments to be passed to the handlerreturn                        : Returns a full list of all file paths created, doesnt return base path as part of thisdativa.tools.aws import S3Csv2ParquetAn easy to use module for converting csv files on s3 to praquet using aws glue jobs.
For S3 access and glue access suitable credentials should be available in '~/.aws/credentials' or the AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY environment variables.S3Csv2ParquetParameters:region - str,
AWS region in which glue job is to be runtemplate_location - str,
S3 bucket Folder in which template scripts are
located or need to be copied.
format s3://bucketname/folder/
it is not clear to those unfamiliar with glue what this is.glue_role - str,
Name of the glue role which need to be assigned to the
Glue Job.max_jobs - int, default 5
Maximum number of jobs the can run concurrently in the queueretry_limit - int, default 3
Maximum number of retries allowed per job on failureconvertParameters:csv_path - str or list of str for multiple files,
s3 location of the csv file
format s3://bucketname/folder/file.csv
Pass a list for multiple filesoutput_folder - str, default set to folder where csv files are located
s3 location at which paraquet file should be copied
format s3://bucketname/folderschema - list of tuples,
If not specified schema is inferred from the file
format [(column1, datatype), (column2, datatype)]
Supported datatypes are boolean, double, float, integer,
long, null, short, stringname - str, default 'parquet_csv_convert'
Name to be assigned to glue joballocated_capacity - int, default 2
The number of AWS Glue data processing units (DPUs) to allocate to this Job.
From 2 to 100 DPUs can be allocateddelete_csv - boolean, default False
If set source csv files are deleted post successful completion of jobseparator - character, default ','
Delimiter character in csv fileswith_header- int, default 1
Specifies whether to treat the first line as a header
Can take values 0 or 1compression - str, default None
If not specified compression is not applied.
Can take values snappy, gzip, and lzopartition_by - list of str, default None
List containing columns to partition data bymode - str, default append
Options include:
overwrite: will remove data from output_folder before writing out
converted file.
append: Will write out to  output_folder without deleting existing
data.
ignore: Silently ignore this operation if data already exists.Examplefromdativa.tools.awsimportS3Csv2Parquet# Initial setupcsv2parquet_obj=S3Csv2Parquet("us-east-1","s3://my-bucket/templatefolder")# Create/update a glue job to convert csv files and execute itcsv2parquet_obj.convert("s3://my-bucket/file_to_be_converted_1.csv")csv2parquet_obj.convert("s3://my-bucket/file_to_be_converted_2.csv")# Wait for completion of jobscsv2parquet_obj.wait_for_completion()dativa.tools.pandas.ShapleyShapley attribution of scores to members of sets.Seemediumorwikifor details on the math. The aim is to apportion scores between the members of a set
responsible for producing that score.Takes two DataFrames as input, one containing impressions from campaigns and the other containing conversions.eg. campaignsviewer_idcampaign_id20B19B12B6A17B12B8B3A18Awhere a campaign_id might be a campaign that an individual has interacted with. Each viewer / campaign
pair is only considered once.eg 2. conversions|viewer_id|
|: --- :|
|14|
|12|
|2|
|3|
|11|these are individuals that have converted. From this a conversion rate is calculated which is used as a score.
That score is apportioned among the campaigns.Typically not all combinations of impressions are present (particularly in where there are lots of
campaigns being run). The missing combinations are assigned a default score, which might be an average conversion rate
or similar. This is passed to the class constructor asdefault_scoreand must be numeric.Example codes=Shapley(df_campaigns,df_conversions,'viewer_id','campaign_id',default_score=0.01)results=s.run()dativa.tools.FileValidationA module to validate the files present in a given directory based on glob pattern matching. The
FileValidation class initialises and checks a set of rules for a file validation, this can then be run for various
paths and glob pattern matches via a method (run_validator). Validation is be based on
various properties - time last modified, size and the number of files which match criteria. Examples on how to use
the file validation are provided below.FileValidation Arguments:timestamp: (tuple or str or None) strings for min and max time (str or None, str or None) in UTC. This is
read in with datetime.strptime, so must use compatible formatsfile_size: (tuple or int or None) min and max size in bytes (int or None, int or None), treats single int
as minimumfile_count: (tuple or int or None) min and max file count, treats single int as minimumtimestamp_format: (str) string for a valid timestamp, specified format for both max and min timestamp,
uses datetime.strptime, hence must be in compatible format.run_validator Arguments:path: (str) path to the files - can either be a path for S3Location or an absolute local file path.  CANNOT HAVE '/' AS THE FINAL CHARACTER.glob_pattern: (str) pattern of glob files to be examined.  CANNOT HAVE '/' AS THE FIRST CHARACTER.consider_correctly_formed_files: (bool) if False, consider malformed files (which do not match validation) for the
file_count argument of FileValidationraise_exceptions: (bool) whether to raise exception when finding files which do not match validation criteriaExample codesThe code can either return a list of files (either all which match the criteria or those which do not) or raise an
exception if any files fail the validation.  This can be applied to either files on S3 or on the local filesystem.The files to be validated are specified via s3fs.glob for S3 (seehttps://s3fs.readthedocs.io/en/latest/) and glob.glob
for local files (seehttps://docs.python.org/3/library/glob.html).Check number of CSVs in a given pathfromdativa.toolsimportFileValidation# ensure there are between 10 and 100 CSVs in your target folder# this command ignores other file extensions (e.g. .tsv files) as they aren't picked up by the glob patternfv=FileValidation(file_count=(10,100))# between 10 and 100 bytes# if validation is not met, raise an exceptionfv.run_validator(path="s3://some-bucket/some-path",glob_pattern="*.csv",raise_exceptions=True)Check that a specific file was last modified on a specific date and is smaller than a given sizefromdativa.toolsimportFileValidationfv=FileValidation(file_size=(None,2*2**10),# 2 KBtimestamp=("2018-12-25","2018-12-25"),# made on 25th Dec 2018timestamp_format="%Y-%m-%d")# if validation is not met, raise an exceptionfv.run_validator(path="s3://some-bucket/some-path",glob_pattern="some-specific-key.csv",raise_exceptions=True)Return file which do not match a certain size and age rangefromdativa.toolsimportFileValidationfromdatetimeimportdatetimefv=FileValidation(file_size=(10,None),# minimum file size of 10 bytes# no minimum time, earliest files made todaytimestamp=(None,datetime.utcnow().strftime("%Y-%m-%d")),timestamp_format="%Y-%m-%d")# return files in dictionarylist_of_bad_files=fv.run_validator(path="/Users/your-name/absolute/file/path",glob_pattern="*",consider_correctly_formed_files=False)Local file paths must beabsolutepaths rather than relative pathsfromdativa.toolsimportFileValidationfv=FileValidation(file_size=(10,10*2**20)# between 10 bytes and 10 GB)# return files which do not match the specified validationlist_of_bad_files=fv.run_validator(path="/Users/your-name/absolute/file/path",glob_pattern="*",consider_correctly_formed_files=False)# return files which match the specified validationlist_of_good_files=fv.run_validator(path="/Users/your-name/absolute/file/path",glob_pattern="*",consider_correctly_formed_files=True)NOTE - both the path must not end with a '/' AND glob_pattern must not start with onefromdativa.toolsimportFileValidationfv=FileValidation(file_size=(10,None),# minimum file size of 10 bytes)# return files in dictionarythis_will_not_run=fv.run_validator(path="/Users/your-name/absolute/file/path/",glob_pattern="/*",consider_correctly_formed_files=False)also_will_not_run=fv.run_validator(path="/Users/your-name/absolute/file/path/",glob_pattern="*",consider_correctly_formed_files=False)again_will_not_run=fv.run_validator(path="/Users/your-name/absolute/file/path",glob_pattern="/*",consider_correctly_formed_files=False)Legacy classesThe modules in the dativatools namespace are legacy only and will be deprecated in future.